|
MLIR-AIE 1.4.6.dev
|
Classes | |
| class | TokenAnalysis |
| class | BlockwriteData |
| struct | SubviewTraceResult |
| struct | ConfigStep |
| struct | AIEResourceStagesOptions |
| Options of aie-resource-stages, named and defaulted as aiecc's flags. More... | |
| struct | ConstStridePolicy |
| struct | SsaStridePolicy |
| struct | BdTemplateFields |
| struct | NdDmaPattern |
| Innermost-first ND access pattern (d0..d3 / repeat), matching the convention used by verifyStridesWraps and NpuDmaMemcpyNdOp verification. More... | |
| struct | TaskBdDecomposition |
| How aie-decompose-large-dma-bd splits a runtime task's descriptor. More... | |
| class | DmaQueueModel |
| Tracks, per DMA channel, the pushes not yet known to have completed along one execution path. More... | |
| struct | SyncCover |
| The channels an aiex.npu.sync syncChannelKey cannot pin down may wait on: its tile range expanded, any value for an operand known only at run time. More... | |
| struct | QueueEffect |
| What one op in a loop body does to a channel queue. More... | |
| struct | DmaCompletionState |
| Completion is distinct from queue occupancy: Task_Queue_Size excludes the running task. More... | |
Typedefs | |
| using | Volume = std::map<int64_t, int64_t> |
| using | ChannelVisits |
| Per-channel analysis visits only the block each push, await or poll reaches through, plus the ops enclosing them, in block order, rather than the whole sequence. | |
Enumerations | |
| enum class | Unbalanced { No , Unsized , Traffic , Excess , PartGranule } |
Functions | |
| std::optional< SubviewTraceResult > | traceSubviewToBlockArgument (Value value) |
| std::optional< unsigned > | getHostBufferArgIndex (BlockArgument arg) |
| unsigned | getHostBufferCount (Block &block) |
| uint64_t | getBdAddressUnit (const AIE::AIETargetModel &targetModel, int col, int row) |
| LogicalResult | emitUpdateBdAddressFromOffsetParameter (OpBuilder &builder, Operation *bdOp, BaseMemRefType bufType, uint64_t registerAddr, const AIE::AIETargetModel &targetModel, int col, int row, std::optional< uint64_t > staticByteAddress=std::nullopt, OpFoldResult lengthField={}) |
| LogicalResult | emitUpdateBdLengthFromParameter (OpBuilder &builder, Operation *bdOp, BaseMemRefType bufType, int64_t lengthUnit, const AIE::AIETargetModel &targetModel, int col, int row, int bdId) |
| bool | getReachableConfigures (Value task, SmallVectorImpl< DMAConfigureTaskOp > &configures) |
| DMAConfigureTaskOp | getUniqueReachableConfigure (Value task) |
| Operation * | getRuntimeDmaTile (Operation *op, SymbolTable &symbols) |
| void | forEachRuntimePinnedBdId (AIE::DeviceOp device, llvm::function_ref< void(Operation *op, Operation *tile, std::optional< int > channel, int32_t id)> pinned) |
| std::optional< int64_t > | staticExecutions (Operation *op) |
| bool | atMost (const Volume &a, int64_t scaleA, const Volume &b, int64_t scaleB) |
| bool | wholeGranules (const Volume &bytes, int64_t granule) |
| Unbalanced | balanceUnder (AIE::ConsumedByAttr proof, const Volume &in, const Volume &out) |
| bool | namesFifo (Operation *op, StringRef fifo, SymbolTableCollection &symbols) |
| LogicalResult | checkResidentsSurvive (AIE::RuntimeSequenceOp seq, ArrayRef< ConfigStep > steps) |
| std::string | scratchpadCommandName (AIE::RuntimeSequenceOp seq) |
| void | emitScratchpadParamsFile (mlir::ModuleOp moduleOp, llvm::raw_ostream &os) |
| void | getHardwareStridesWraps (const AIE::AIETargetModel &targetModel, mlir::Operation *op, mlir::BaseMemRefType referencedBufType, llvm::SmallVector< int64_t, 4 > inputSizes, llvm::SmallVector< int64_t, 4 > inputStrides, llvm::SmallVector< int64_t, 4 > &sizes, llvm::SmallVector< int64_t, 4 > &strides) |
| mlir::LogicalResult | verifyStridesWraps (mlir::Operation *forOp, llvm::function_ref< mlir::InFlightDiagnostic()> emitError, mlir::BaseMemRefType referencedBufType, int tileCol, int tileRow, llvm::SmallVector< int64_t, 4 > inputSizes, llvm::SmallVector< int64_t, 4 > inputStrides, llvm::SmallVector< int64_t, 4 > hardwareSizes, llvm::SmallVector< int64_t, 4 > hardwareStrides, bool skipTransformationChecks=false) |
| bool | isLinearTransfer (llvm::ArrayRef< int64_t > sizes, llvm::ArrayRef< int64_t > strides) |
| bool | isContiguousTransfer (llvm::ArrayRef< int64_t > sizes, llvm::ArrayRef< int64_t > strides) |
| bool | isLinearModeTransfer (const AIE::AIETargetModel &targetModel, int col, int row, llvm::ArrayRef< int64_t > sizes, llvm::ArrayRef< int64_t > strides) |
| std::optional< uint32_t > | getConstantIntOperand (mlir::Value v) |
| std::optional< uint64_t > | getConstantInt64Operand (mlir::Value v) |
| mlir::Value | createConstantI32 (mlir::OpBuilder &builder, mlir::Location loc, uint32_t value) |
| mlir::Value | createConstantArgPlus (mlir::OpBuilder &builder, mlir::Location loc, uint64_t value) |
| mlir::Operation * | findShimDmaStart (AIE::DeviceOp device) |
| std::unique_ptr< mlir::OperationPass< AIE::DeviceOp > > | createAIECreateCoresPass () |
| std::unique_ptr< mlir::OperationPass< AIE::DeviceOp > > | createAIECreateLocksPass () |
| std::unique_ptr< mlir::OperationPass< AIE::DeviceOp > > | createAIELowerMemcpyPass () |
| std::unique_ptr< mlir::OperationPass< AIE::DeviceOp > > | createAIELowerMulticastPass () |
| std::unique_ptr< mlir::OperationPass< AIE::DeviceOp > > | createAIEBroadcastPacketPass () |
| std::unique_ptr< mlir::OperationPass< AIE::DeviceOp > > | createAIEDmaToNpuPass () |
| std::unique_ptr< mlir::OperationPass< AIE::DeviceOp > > | createAIEDmaToNpuPass (const AIEDmaToNpuOptions &options) |
| std::unique_ptr< mlir::OperationPass< AIE::DeviceOp > > | createAIEDecomposeLargeDmaBdPass () |
| std::unique_ptr< mlir::OperationPass< mlir::ModuleOp > > | createAIENpuToCertPass () |
| std::unique_ptr< mlir::OperationPass< AIE::DeviceOp > > | createAIECertPagesPass () |
| std::unique_ptr< mlir::OperationPass< AIE::DeviceOp > > | createAIECertVerifyPass () |
| std::unique_ptr< mlir::OperationPass< mlir::ModuleOp > > | createAIEXToStandardPass () |
| std::unique_ptr< mlir::OperationPass< mlir::ModuleOp > > | createAIESCFToControlFlowPass () |
| std::unique_ptr< mlir::OperationPass< AIE::DeviceOp > > | createAIEMaterializeBDChainsPass () |
| std::unique_ptr< mlir::OperationPass< mlir::ModuleOp > > | createAIEMaterializeRuntimeSequencesPass () |
| std::unique_ptr< mlir::OperationPass< mlir::ModuleOp > > | createAIEFuseTraceBuffersPass () |
| std::unique_ptr< mlir::OperationPass< AIE::DeviceOp > > | createAIEResolveAddressPatchBuffersPass () |
| std::unique_ptr< mlir::OperationPass< AIE::DeviceOp > > | createAIEAssignRuntimeSequenceBDIDsPass () |
| std::unique_ptr< mlir::OperationPass< AIE::DeviceOp > > | createAIEAssignRuntimeSequenceBDIDsPass (const AIEAssignRuntimeSequenceBDIDsOptions &options) |
| std::unique_ptr< mlir::OperationPass< AIE::DeviceOp > > | createAIESplitLongRepeatsPass () |
| std::unique_ptr< mlir::OperationPass< AIE::DeviceOp > > | createAIEReserveRuntimeBDIDsPass () |
| std::unique_ptr< mlir::OperationPass< AIE::DeviceOp > > | createAIEUnrollRuntimeSequenceLoopsPass () |
| std::unique_ptr< mlir::OperationPass< AIE::DeviceOp > > | createAIELowerDynamicBDPoolPass () |
| std::unique_ptr< mlir::OperationPass< AIE::DeviceOp > > | createAIELowerDynamicBDPoolPass (const AIELowerDynamicBDPoolOptions &options) |
| std::unique_ptr< mlir::OperationPass< AIE::DeviceOp > > | createAIEDMATasksToNPUPass () |
| std::unique_ptr< mlir::OperationPass< AIE::DeviceOp > > | createAIESubstituteShimDMAAllocationsPass () |
| std::unique_ptr< mlir::OperationPass< AIE::DeviceOp > > | createAIECtrlPacketToDmaPass () |
| std::unique_ptr< mlir::OperationPass< AIE::DeviceOp > > | createAIECtrlPacketInferTilesPass () |
| std::unique_ptr< mlir::OperationPass< AIE::DeviceOp > > | createAIELowerSetLockPass () |
| std::unique_ptr< mlir::OperationPass< AIE::DeviceOp > > | createAIELowerDmaChannelResetPass () |
| std::unique_ptr< mlir::OperationPass< AIE::DeviceOp > > | createAIELowerCoreResetPass () |
| std::unique_ptr< mlir::OperationPass< AIE::DeviceOp > > | createAIEVerifyRuntimeRearmPass () |
| std::unique_ptr< mlir::OperationPass< AIE::DeviceOp > > | createAIETransformBfpTypesPass () |
| std::unique_ptr< mlir::OperationPass< AIE::DeviceOp > > | createAIETxnToControlPacketPass () |
| std::unique_ptr< mlir::OperationPass< AIE::DeviceOp > > | createAIELegalizeControlPacketPass () |
| std::unique_ptr< mlir::OperationPass< mlir::ModuleOp > > | createAIEExpandLoadPdiPass () |
| std::unique_ptr< mlir::OperationPass< AIE::DeviceOp > > | createAIEXInlineTraceConfigPass () |
| std::unique_ptr< mlir::OperationPass< mlir::ModuleOp > > | createAIELowerScratchpadParametersPass () |
| std::unique_ptr< mlir::OperationPass< mlir::ModuleOp > > | createAIELowerScratchpadParametersPass (AIELowerScratchpadParametersOptions options) |
| void | addPlacementStage (mlir::OpPassManager &pm, const AIE::AIEPlaceTilesOptions &options) |
| aiecc's stages that give a design its tiles, locks, BD IDs, buffers and routes, in its order. | |
| void | addTraceStage (mlir::OpPassManager &pm) |
| void | addLoweringStage (mlir::OpPassManager &pm, bool dynamicObjFifos, bool packetSwObjFifos, bool ctrlPktOverlay, bool loadPdiToCtrlPkt, bool skipObjectFifoVerify, llvm::ArrayRef< int > coverColumns={}, bool adaptiveOverlay=true, bool checkQueueSpace=true) |
| Affine lowering through aie-prepare-buffers. | |
| void | addBufferAddressStage (mlir::OpPassManager &pm, int64_t placementBudget) |
| void | addRoutingStage (mlir::OpPassManager &pm, bool allowDeadlockProne) |
| void | registerAIEResourceStagesPipeline () |
| int64_t | bdGranuleDivisor (uint64_t elemWidth, uint32_t addressGranularity) |
| bool | isConstMultipleOfGranule (int64_t value, uint64_t elemWidth, uint32_t addressGranularity) |
| mlir::LogicalResult | verifyConstBdRealizability (mlir::Operation *op, llvm::ArrayRef< mlir::OpFoldResult > sizes, llvm::ArrayRef< mlir::OpFoldResult > strides, uint64_t elemWidth, uint32_t gran) |
| template<typename Policy> | |
| void | encodeHardwareStridesWraps (Policy &p, uint64_t elemWidth, uint32_t addressGranularity, typename Policy::V inputSizes[4], typename Policy::V inputStrides[4], typename Policy::V sizes[4], typename Policy::V strides[4]) |
| mlir::LogicalResult | emitRuntimeCheck (mlir::OpBuilder &builder, mlir::Location loc, mlir::Value cond, const llvm::Twine &message) |
| void | eraseDeadArith (mlir::Operation *root) |
| mlir::Value | getAsValue (mlir::OpBuilder &builder, mlir::Location loc, mlir::OpFoldResult ofr, mlir::Type intType) |
| mlir::Value | getAsI64 (mlir::OpBuilder &builder, mlir::Location loc, mlir::OpFoldResult ofr) |
| mlir::Value | buildArgPlusValue (mlir::OpBuilder &builder, mlir::Location loc, llvm::ArrayRef< mlir::OpFoldResult > elementOffsets, llvm::ArrayRef< mlir::OpFoldResult > strides, int64_t elemWidthBytes, int64_t baseByteOffset, uint32_t granuleBytes) |
| mlir::LogicalResult | guardWithinHostBuffer (mlir::OpBuilder &builder, mlir::Location loc, mlir::BaseMemRefType hostBufferType, int64_t baseByteOffset, int64_t elemWidthBytes, llvm::ArrayRef< mlir::OpFoldResult > offsets, llvm::ArrayRef< mlir::OpFoldResult > offsetStrides, llvm::ArrayRef< mlir::OpFoldResult > sizes, llvm::ArrayRef< mlir::OpFoldResult > strides) |
| mlir::Value | buildBdWord (mlir::OpBuilder &builder, mlir::Location loc, llvm::ArrayRef< std::tuple< mlir::Value, uint32_t, uint32_t > > fields) |
| mlir::Value | getBdRegisterBase (mlir::OpBuilder &builder, mlir::Location loc, const xilinx::AIE::AIETargetModel &targetModel, int tileCol, int tileRow, mlir::OpFoldResult bdId) |
| mlir::LogicalResult | buildBdWords (mlir::OpBuilder &builder, mlir::Location loc, const xilinx::AIE::AIETargetModel &targetModel, int tileCol, int tileRow, const BdTemplateFields &fields, llvm::ArrayRef< mlir::OpFoldResult > mixedSizes, llvm::ArrayRef< mlir::OpFoldResult > mixedStrides, uint64_t elemWidth, uint32_t burstLength, uint32_t axcache, mlir::OpFoldResult lenElems, mlir::Value &repeatCountOut, llvm::SmallVectorImpl< mlir::Value > &wordsOut) |
| bool | patternPassesVerification (mlir::Operation *forOp, mlir::BaseMemRefType referencedBufType, const xilinx::AIE::AIETargetModel &tm, int tileCol, int tileRow, const NdDmaPattern &pattern, bool skipTransformationChecks=false) |
| Returns true when the pattern passes verifyStridesWraps for the given tile and memref element type. | |
| bool | isDecomposableNdDmaPattern (mlir::Operation *forOp, mlir::BaseMemRefType referencedBufType, const xilinx::AIE::AIETargetModel &targetModel, int tileCol, int tileRow, llvm::ArrayRef< int64_t > offsetsInnermostFirst, llvm::ArrayRef< int64_t > sizesInnermostFirst, llvm::ArrayRef< int64_t > stridesInnermostFirst) |
| True when the pattern is fully constant, non-contiguous, currently illegal, and decomposeNdDmaPattern would produce at least one legal replacement. | |
| mlir::FailureOr< llvm::SmallVector< NdDmaPattern > > | decomposeNdDmaPattern (mlir::Operation *forOp, mlir::BaseMemRefType referencedBufType, const NdDmaPattern &pattern, const xilinx::AIE::AIETargetModel &targetModel, int tileCol, int tileRow) |
| Decompose an illegal pattern into one or more legal sub-patterns that move the same data. | |
| mlir::FailureOr< TaskBdDecomposition > | decomposeTaskBdPattern (xilinx::AIE::DMABDOp bd, const xilinx::AIE::AIETargetModel &targetModel, int tileCol, int tileRow) |
| Plan the split of bd, a runtime task's descriptor, on tile (tileCol, tileRow). | |
| std::optional< DmaQueueModel::ChannelKey > | syncChannelKey (NpuSyncOp sync) |
| The channel an aiex.npu.sync retires on, or nullopt when it cannot be pinned down. | |
| SyncCover | syncCover (NpuSyncOp sync) |
| void | awaitSync (DmaQueueModel &queue, NpuSyncOp sync) |
| Retire what an aiex.npu.sync proves has drained. | |
| mlir::InFlightDiagnostic | emitQueueOverflowWarning (mlir::Operation *op, int col, int row, AIE::DMAChannelDir dir, uint32_t channel, uint32_t depth, size_t outstanding) |
| Shared wording for the two lowering paths, so the explanation does not depend on whether the transfer came from dma_start_task or npu.dma_memcpy_nd. | |
| mlir::InFlightDiagnostic | emitUnsafeBdReuseWarning (mlir::Operation *freeOp, int col, int row, AIE::DMAChannelDir dir, uint32_t channel) |
| Wording for a dma_free_task that releases a buffer descriptor id the runtime pool could hand straight back out before hardware has actually finished the freed transfer – the rolled-loop counterpart to aie-assign-runtime-sequence-bd-ids's unsafe-BD-reuse warning. | |
| std::optional< std::pair< uint32_t, uint32_t > > | queueSpacePoll (const AIE::AIETargetModel &tm, int col, int row, uint32_t channel, AIE::DMAChannelDir dir, uint32_t depth) |
| Address and mask for a poll that waits for a free task-queue slot. | |
| mlir::LogicalResult | insertQueueSpaceWait (mlir::Operation *before, const AIE::AIETargetModel &tm, int col, int row, uint32_t channel, AIE::DMAChannelDir dir, uint32_t depth) |
| void | guardQueueOverflow (DmaQueueModel &queue, mlir::Operation *push, const AIE::AIETargetModel &tm, const DmaQueueModel::ChannelKey &key, uint32_t depth, bool enforce) |
| Handle a push on key that would land on a full queue: poll where one can be emitted, warn where it cannot (see insertQueueSpaceWait above for when that is). | |
| QueueEffect | rawQueueEffect (mlir::Operation *op, mlir::SymbolTable &symbols) |
| What a push or an await that names no task does to its channel: an aiex.npu.push_queue; an aiex.npu.dma_memcpy_nd through the placed shim allocation its metadata names, which always issues a token on S2MM (its lowering sets one); an aiex.npu.sync syncChannelKey pins down; an aiex.npu.dma_wait on a placed shim allocation. | |
| void | recordChannelVisit (mlir::Region &body, mlir::Operation *op, ChannelVisits &blocks) |
| Record op and each operation enclosing it up to body's own op into blocks, once per block: an ancestor chain's visited block repeats until the chain's op there changes. | |
| template<typename EffectFn> | |
| void | guardSequenceQueueDepth (mlir::Region &body, DmaQueueModel &queue, const AIE::AIETargetModel &tm, bool enforce, bool warnUnsafeBdReuse, EffectFn effectOf) |
| Guard every push in a runtime sequence that can land on a full queue, and (when warnUnsafeBdReuse) warn at every Free effect whose own push is not yet proven complete along some reachable path. | |
| template<typename EffectFn> | |
| mlir::LogicalResult | checkTokensConsumed (AIE::RuntimeSequenceOp sequence, const std::set< DmaQueueModel::ChannelKey > &awaited, llvm::ArrayRef< SyncCover > mayAwait, EffectFn effectOf) |
| Check that sequence ends, on every path, with no task-completion token outstanding on a channel some sequence of the device awaits: one in awaited, or one a sync in mayAwait may wait on. | |
| LogicalResult | emitRuntimeCheck (OpBuilder &builder, Location loc, Value cond, const Twine &message) |
| void | eraseDeadArith (Operation *root) |
| Value | getAsI64 (OpBuilder &builder, Location loc, OpFoldResult ofr) |
| Value | getAsValue (OpBuilder &builder, Location loc, OpFoldResult ofr, Type intType) |
| Value | buildArgPlusValue (OpBuilder &builder, Location loc, ArrayRef< OpFoldResult > elementOffsets, ArrayRef< OpFoldResult > strides, int64_t elemWidthBytes, int64_t baseByteOffset, uint32_t granuleBytes) |
| LogicalResult | guardWithinHostBuffer (OpBuilder &builder, Location loc, BaseMemRefType hostBufferType, int64_t baseByteOffset, int64_t elemWidthBytes, ArrayRef< OpFoldResult > offsets, ArrayRef< OpFoldResult > offsetStrides, ArrayRef< OpFoldResult > sizes, ArrayRef< OpFoldResult > strides) |
| Value | buildBdWord (OpBuilder &builder, Location loc, ArrayRef< std::tuple< Value, uint32_t, uint32_t > > fields) |
| Value | getBdRegisterBase (OpBuilder &builder, Location loc, const AIE::AIETargetModel &targetModel, int tileCol, int tileRow, OpFoldResult bdId) |
| LogicalResult | buildBdWords (OpBuilder &builder, Location loc, const AIE::AIETargetModel &targetModel, int tileCol, int tileRow, const BdTemplateFields &f, ArrayRef< OpFoldResult > mixedSizes, ArrayRef< OpFoldResult > mixedStrides, uint64_t elemWidth, uint32_t burstLength, uint32_t axcache, OpFoldResult lenElems, Value &repeatCountOut, SmallVectorImpl< Value > &wordsOut) |
Variables | |
| constexpr llvm::StringLiteral | initEntrySequenceName = "reconfig_init" |
| Name of the runtime sequence aie-expand-load-pdi{init-entry} adds. | |
| constexpr llvm::StringLiteral | queueDiagnosedAttr |
Per-channel analysis visits only the block each push, await or poll reaches through, plus the ops enclosing them, in block order, rather than the whole sequence.
Definition at line 526 of file DmaQueueModel.h.
| using xilinx::AIEX::Volume = std::map<int64_t, int64_t> |
Definition at line 146 of file AIEUtils.h.
|
strong |
| Enumerator | |
|---|---|
| No | |
| Unsized | |
| Traffic | |
| Excess | |
| PartGranule | |
Definition at line 157 of file AIEUtils.h.
| void xilinx::AIEX::addBufferAddressStage | ( | mlir::OpPassManager & | pm, |
| int64_t | placementBudget ) |
Referenced by registerAIEResourceStagesPipeline().
| void xilinx::AIEX::addLoweringStage | ( | mlir::OpPassManager & | pm, |
| bool | dynamicObjFifos, | ||
| bool | packetSwObjFifos, | ||
| bool | ctrlPktOverlay, | ||
| bool | loadPdiToCtrlPkt, | ||
| bool | skipObjectFifoVerify, | ||
| llvm::ArrayRef< int > | coverColumns = {}, | ||
| bool | adaptiveOverlay = true, | ||
| bool | checkQueueSpace = true ) |
Affine lowering through aie-prepare-buffers.
Referenced by registerAIEResourceStagesPipeline().
| void xilinx::AIEX::addPlacementStage | ( | mlir::OpPassManager & | pm, |
| const AIE::AIEPlaceTilesOptions & | options ) |
aiecc's stages that give a design its tiles, locks, BD IDs, buffers and routes, in its order.
aiecc runs them as separate steps; the aie-resource-stages pipeline runs them back to back. Each anchors on the module.
Referenced by registerAIEResourceStagesPipeline().
| void xilinx::AIEX::addRoutingStage | ( | mlir::OpPassManager & | pm, |
| bool | allowDeadlockProne ) |
Referenced by registerAIEResourceStagesPipeline().
| void xilinx::AIEX::addTraceStage | ( | mlir::OpPassManager & | pm | ) |
Referenced by registerAIEResourceStagesPipeline().
Definition at line 354 of file AIEUtils.cpp.
Referenced by balanceUnder().
|
inline |
Retire what an aiex.npu.sync proves has drained.
This is the lowered form of npu.dma_wait and the same hardware event, so it retires by the same rule; sequences written or lowered down to raw syncs would otherwise look like they never drain a channel at all.
Definition at line 221 of file DmaQueueModel.h.
References xilinx::AIEX::DmaQueueModel::awaitToken(), and syncChannelKey().
| AIEX::Unbalanced xilinx::AIEX::balanceUnder | ( | AIE::ConsumedByAttr | proof, |
| const Volume & | in, | ||
| const Volume & | out ) |
Definition at line 377 of file AIEUtils.cpp.
References atMost(), Excess, No, PartGranule, and wholeGranules().
|
inline |
Definition at line 58 of file BdLowering.h.
| mlir::Value xilinx::AIEX::buildArgPlusValue | ( | mlir::OpBuilder & | builder, |
| mlir::Location | loc, | ||
| llvm::ArrayRef< mlir::OpFoldResult > | elementOffsets, | ||
| llvm::ArrayRef< mlir::OpFoldResult > | strides, | ||
| int64_t | elemWidthBytes, | ||
| int64_t | baseByteOffset, | ||
| uint32_t | granuleBytes ) |
Referenced by AIEDMATasksToNPUPass::setAddressForSingleBD().
| Value xilinx::AIEX::buildArgPlusValue | ( | OpBuilder & | builder, |
| Location | loc, | ||
| ArrayRef< OpFoldResult > | elementOffsets, | ||
| ArrayRef< OpFoldResult > | strides, | ||
| int64_t | elemWidthBytes, | ||
| int64_t | baseByteOffset, | ||
| uint32_t | granuleBytes ) |
Definition at line 190 of file BdLowering.cpp.
References getAsI64().
| mlir::Value xilinx::AIEX::buildBdWord | ( | mlir::OpBuilder & | builder, |
| mlir::Location | loc, | ||
| llvm::ArrayRef< std::tuple< mlir::Value, uint32_t, uint32_t > > | fields ) |
| Value xilinx::AIEX::buildBdWord | ( | OpBuilder & | builder, |
| Location | loc, | ||
| ArrayRef< std::tuple< Value, uint32_t, uint32_t > > | fields ) |
Definition at line 307 of file BdLowering.cpp.
| mlir::LogicalResult xilinx::AIEX::buildBdWords | ( | mlir::OpBuilder & | builder, |
| mlir::Location | loc, | ||
| const xilinx::AIE::AIETargetModel & | targetModel, | ||
| int | tileCol, | ||
| int | tileRow, | ||
| const BdTemplateFields & | fields, | ||
| llvm::ArrayRef< mlir::OpFoldResult > | mixedSizes, | ||
| llvm::ArrayRef< mlir::OpFoldResult > | mixedStrides, | ||
| uint64_t | elemWidth, | ||
| uint32_t | burstLength, | ||
| uint32_t | axcache, | ||
| mlir::OpFoldResult | lenElems, | ||
| mlir::Value & | repeatCountOut, | ||
| llvm::SmallVectorImpl< mlir::Value > & | wordsOut ) |
Referenced by AIEDMATasksToNPUPass::rewriteSingleBDDynamic().
| LogicalResult xilinx::AIEX::buildBdWords | ( | OpBuilder & | builder, |
| Location | loc, | ||
| const AIE::AIETargetModel & | targetModel, | ||
| int | tileCol, | ||
| int | tileRow, | ||
| const BdTemplateFields & | f, | ||
| ArrayRef< OpFoldResult > | mixedSizes, | ||
| ArrayRef< OpFoldResult > | mixedStrides, | ||
| uint64_t | elemWidth, | ||
| uint32_t | burstLength, | ||
| uint32_t | axcache, | ||
| OpFoldResult | lenElems, | ||
| Value & | repeatCountOut, | ||
| SmallVectorImpl< Value > & | wordsOut ) |
Definition at line 645 of file BdLowering.cpp.
References xilinx::AIE::AIETargetModel::getDmaBdLayout().
| LogicalResult xilinx::AIEX::checkResidentsSurvive | ( | AIE::RuntimeSequenceOp | seq, |
| ArrayRef< ConfigStep > | steps ) |
Definition at line 396 of file AIEUtils.cpp.
References xilinx::AIEX::ConfigStep::configures, xilinx::AIEX::ConfigStep::load, xilinx::AIEX::ConfigStep::op, and xilinx::AIEX::ConfigStep::resets.
|
inline |
Check that sequence ends, on every path, with no task-completion token outstanding on a channel some sequence of the device awaits: one in awaited, or one a sync in mayAwait may wait on.
A token left at the end outlives the command: the next command's await on that channel takes it and returns before its own transfer has finished (measured on Strix, NPU firmware 1.1.2.64), and every completion the compiler derives from an await assumes the channel started with none. A token on a channel nothing awaits is never taken. An await consumes the oldest token, so only the count matters. A loop with an unknown trip count must leave the count where it found it, or what remains grows with the trip count. Regions the check cannot follow are not diagnosed, nor channels a sync of the sequence's own may have waited on.
Definition at line 958 of file DmaQueueModel.h.
References xilinx::AIEX::QueueEffect::Await, xilinx::AIEX::SyncCover::covers(), xilinx::AIEX::QueueEffect::key, xilinx::AIEX::QueueEffect::kind, xilinx::AIEX::QueueEffect::Push, recordChannelVisit(), syncChannelKey(), and syncCover().
| std::unique_ptr< OperationPass< AIE::DeviceOp > > xilinx::AIEX::createAIEAssignRuntimeSequenceBDIDsPass | ( | ) |
Definition at line 2151 of file AIEAssignRuntimeSequenceBDIDs.cpp.
| std::unique_ptr< OperationPass< AIE::DeviceOp > > xilinx::AIEX::createAIEAssignRuntimeSequenceBDIDsPass | ( | const AIEAssignRuntimeSequenceBDIDsOptions & | options | ) |
Definition at line 2156 of file AIEAssignRuntimeSequenceBDIDs.cpp.
| std::unique_ptr< OperationPass< DeviceOp > > xilinx::AIEX::createAIEBroadcastPacketPass | ( | ) |
Definition at line 104 of file AIECreateBroadcastPacket.cpp.
| std::unique_ptr< OperationPass< AIE::DeviceOp > > xilinx::AIEX::createAIECertPagesPass | ( | ) |
Definition at line 2515 of file AIENpuToCert.cpp.
| std::unique_ptr< OperationPass< AIE::DeviceOp > > xilinx::AIEX::createAIECertVerifyPass | ( | ) |
Definition at line 362 of file AIECertVerify.cpp.
| std::unique_ptr< OperationPass< DeviceOp > > xilinx::AIEX::createAIECreateCoresPass | ( | ) |
Definition at line 255 of file AIECreateCores.cpp.
| std::unique_ptr< OperationPass< DeviceOp > > xilinx::AIEX::createAIECreateLocksPass | ( | ) |
Definition at line 230 of file AIECreateLocks.cpp.
| std::unique_ptr< OperationPass< DeviceOp > > xilinx::AIEX::createAIECtrlPacketInferTilesPass | ( | ) |
Definition at line 228 of file AIECtrlPacketToDma.cpp.
| std::unique_ptr< OperationPass< DeviceOp > > xilinx::AIEX::createAIECtrlPacketToDmaPass | ( | ) |
Definition at line 231 of file AIECtrlPacketToDma.cpp.
| std::unique_ptr< OperationPass< AIE::DeviceOp > > xilinx::AIEX::createAIEDecomposeLargeDmaBdPass | ( | ) |
Definition at line 862 of file AIEDecomposeLargeDmaBd.cpp.
| std::unique_ptr< OperationPass< AIE::DeviceOp > > xilinx::AIEX::createAIEDMATasksToNPUPass | ( | ) |
Definition at line 1360 of file AIEDMATasksToNPU.cpp.
| std::unique_ptr< OperationPass< AIE::DeviceOp > > xilinx::AIEX::createAIEDmaToNpuPass | ( | ) |
Definition at line 985 of file AIEDmaToNpu.cpp.
| std::unique_ptr< OperationPass< AIE::DeviceOp > > xilinx::AIEX::createAIEDmaToNpuPass | ( | const AIEDmaToNpuOptions & | options | ) |
Definition at line 990 of file AIEDmaToNpu.cpp.
| std::unique_ptr< mlir::OperationPass< mlir::ModuleOp > > xilinx::AIEX::createAIEExpandLoadPdiPass | ( | ) |
Definition at line 574 of file AIEExpandLoadPdi.cpp.
| std::unique_ptr< OperationPass< ModuleOp > > xilinx::AIEX::createAIEFuseTraceBuffersPass | ( | ) |
Definition at line 205 of file AIEFuseTraceBuffers.cpp.
| std::unique_ptr< OperationPass< AIE::DeviceOp > > xilinx::AIEX::createAIELegalizeControlPacketPass | ( | ) |
Definition at line 171 of file AIETxnToControlPacket.cpp.
| std::unique_ptr< OperationPass< DeviceOp > > xilinx::AIEX::createAIELowerCoreResetPass | ( | ) |
Definition at line 116 of file AIELowerCoreReset.cpp.
| std::unique_ptr< OperationPass< DeviceOp > > xilinx::AIEX::createAIELowerDmaChannelResetPass | ( | ) |
Definition at line 305 of file AIELowerDmaChannelReset.cpp.
| std::unique_ptr< OperationPass< AIE::DeviceOp > > xilinx::AIEX::createAIELowerDynamicBDPoolPass | ( | ) |
Definition at line 832 of file AIELowerDynamicBDPool.cpp.
| std::unique_ptr< OperationPass< AIE::DeviceOp > > xilinx::AIEX::createAIELowerDynamicBDPoolPass | ( | const AIELowerDynamicBDPoolOptions & | options | ) |
Definition at line 837 of file AIELowerDynamicBDPool.cpp.
| std::unique_ptr< OperationPass< AIE::DeviceOp > > xilinx::AIEX::createAIELowerMemcpyPass | ( | ) |
Definition at line 151 of file AIELowerMemcpy.cpp.
| std::unique_ptr< OperationPass< DeviceOp > > xilinx::AIEX::createAIELowerMulticastPass | ( | ) |
Definition at line 85 of file AIELowerMulticast.cpp.
| std::unique_ptr< OperationPass< ModuleOp > > xilinx::AIEX::createAIELowerScratchpadParametersPass | ( | ) |
Definition at line 1147 of file AIELowerScratchpadParameters.cpp.
| std::unique_ptr< OperationPass< ModuleOp > > xilinx::AIEX::createAIELowerScratchpadParametersPass | ( | AIELowerScratchpadParametersOptions | options | ) |
Definition at line 1152 of file AIELowerScratchpadParameters.cpp.
| std::unique_ptr< OperationPass< DeviceOp > > xilinx::AIEX::createAIELowerSetLockPass | ( | ) |
Definition at line 101 of file AIELowerSetLock.cpp.
| std::unique_ptr< OperationPass< AIE::DeviceOp > > xilinx::AIEX::createAIEMaterializeBDChainsPass | ( | ) |
Definition at line 159 of file AIEMaterializeBDChains.cpp.
| std::unique_ptr< OperationPass< ModuleOp > > xilinx::AIEX::createAIEMaterializeRuntimeSequencesPass | ( | ) |
Definition at line 870 of file AIEMaterializeRuntimeSequences.cpp.
| std::unique_ptr< OperationPass< mlir::ModuleOp > > xilinx::AIEX::createAIENpuToCertPass | ( | ) |
Definition at line 2511 of file AIENpuToCert.cpp.
| std::unique_ptr< mlir::OperationPass< AIE::DeviceOp > > xilinx::AIEX::createAIEReserveRuntimeBDIDsPass | ( | ) |
Definition at line 63 of file AIEReserveRuntimeBDIDs.cpp.
| std::unique_ptr< OperationPass< AIE::DeviceOp > > xilinx::AIEX::createAIEResolveAddressPatchBuffersPass | ( | ) |
Definition at line 113 of file AIEResolveAddressPatchBuffers.cpp.
| std::unique_ptr< mlir::OperationPass< mlir::ModuleOp > > xilinx::AIEX::createAIESCFToControlFlowPass | ( | ) |
Definition at line 83 of file AIESCFToControlFlow.cpp.
| std::unique_ptr< mlir::OperationPass< AIE::DeviceOp > > xilinx::AIEX::createAIESplitLongRepeatsPass | ( | ) |
Definition at line 111 of file AIESplitLongRepeats.cpp.
| std::unique_ptr< OperationPass< AIE::DeviceOp > > xilinx::AIEX::createAIESubstituteShimDMAAllocationsPass | ( | ) |
Definition at line 79 of file AIESubstituteShimDMAAllocations.cpp.
| std::unique_ptr< OperationPass< DeviceOp > > xilinx::AIEX::createAIETransformBfpTypesPass | ( | ) |
Definition at line 217 of file AIETransformBfpTypes.cpp.
| std::unique_ptr< OperationPass< AIE::DeviceOp > > xilinx::AIEX::createAIETxnToControlPacketPass | ( | ) |
Definition at line 166 of file AIETxnToControlPacket.cpp.
| std::unique_ptr< mlir::OperationPass< AIE::DeviceOp > > xilinx::AIEX::createAIEUnrollRuntimeSequenceLoopsPass | ( | ) |
Definition at line 70 of file AIEUnrollRuntimeSequenceLoops.cpp.
| std::unique_ptr< OperationPass< DeviceOp > > xilinx::AIEX::createAIEVerifyRuntimeRearmPass | ( | ) |
Definition at line 155 of file AIEVerifyRuntimeRearm.cpp.
| std::unique_ptr< OperationPass< AIE::DeviceOp > > xilinx::AIEX::createAIEXInlineTraceConfigPass | ( | ) |
Definition at line 127 of file AIEInlineTraceConfig.cpp.
| std::unique_ptr< OperationPass< ModuleOp > > xilinx::AIEX::createAIEXToStandardPass | ( | ) |
Definition at line 74 of file AIEXToStandard.cpp.
| mlir::Value xilinx::AIEX::createConstantArgPlus | ( | mlir::OpBuilder & | builder, |
| mlir::Location | loc, | ||
| uint64_t | value ) |
Definition at line 1051 of file AIEXDialect.cpp.
References createConstantI32().
| mlir::Value xilinx::AIEX::createConstantI32 | ( | mlir::OpBuilder & | builder, |
| mlir::Location | loc, | ||
| uint32_t | value ) |
Definition at line 1045 of file AIEXDialect.cpp.
Referenced by createConstantArgPlus(), AIEDMATasksToNPUPass::emitOutOfOrderChannelEnable(), AIEDMATasksToNPUPass::gatherBdTemplateFields(), getBdRegisterBase(), CoreResetToMaskWrite32Pattern::matchAndRewrite(), DMAAwaitTaskOpPattern::matchAndRewrite(), DmaChannelResetToMaskWrite32Pattern::matchAndRewrite(), SetLockToWrite32Pattern::matchAndRewrite(), AIECtrlPacketToDmaPass::runOnOperation(), and AIEDMATasksToNPUPass::setAddressForSingleBD().
| mlir::FailureOr< llvm::SmallVector< NdDmaPattern > > xilinx::AIEX::decomposeNdDmaPattern | ( | mlir::Operation * | forOp, |
| mlir::BaseMemRefType | referencedBufType, | ||
| const NdDmaPattern & | pattern, | ||
| const xilinx::AIE::AIETargetModel & | targetModel, | ||
| int | tileCol, | ||
| int | tileRow ) |
Decompose an illegal pattern into one or more legal sub-patterns that move the same data.
Prefers merging contiguous dimensions, then dimension factoring (single-op results); falls back to slicing (multiple ops). Returns failure when no legal decomposition into at most kMaxNdDmaPieces patterns exists.
A contiguous pattern is illegal only if its iteration dimension (d3) is longer than a BD's, and is sliced along it.
A pattern with more than kNdDmaDims dimensions drops size-one dimensions past d0, innermost first, merges the iteration dimensions past d2 where one continues the next, and peels any left past d3 into one pattern per index, outermost slowest. Element order is kept; the size of an execution is not, since a dropped dimension can move an iteration dimension inside and a peeled pattern executes a share of the original's.
| mlir::FailureOr< TaskBdDecomposition > xilinx::AIEX::decomposeTaskBdPattern | ( | xilinx::AIE::DMABDOp | bd, |
| const xilinx::AIE::AIETargetModel & | targetModel, | ||
| int | tileCol, | ||
| int | tileRow ) |
Plan the split of bd, a runtime task's descriptor, on tile (tileCol, tileRow).
Fails when bd has padding or a non-constant offset, length, size or stride, which the pass leaves whole.
|
inline |
Shared wording for the two lowering paths, so the explanation does not depend on whether the transfer came from dma_start_task or npu.dma_memcpy_nd.
Definition at line 230 of file DmaQueueModel.h.
Referenced by guardQueueOverflow().
| mlir::LogicalResult xilinx::AIEX::emitRuntimeCheck | ( | mlir::OpBuilder & | builder, |
| mlir::Location | loc, | ||
| mlir::Value | cond, | ||
| const llvm::Twine & | message ) |
| LogicalResult xilinx::AIEX::emitRuntimeCheck | ( | OpBuilder & | builder, |
| Location | loc, | ||
| Value | cond, | ||
| const Twine & | message ) |
Definition at line 76 of file BdLowering.cpp.
| void xilinx::AIEX::emitScratchpadParamsFile | ( | mlir::ModuleOp | moduleOp, |
| llvm::raw_ostream & | os ) |
|
inline |
Wording for a dma_free_task that releases a buffer descriptor id the runtime pool could hand straight back out before hardware has actually finished the freed transfer – the rolled-loop counterpart to aie-assign-runtime-sequence-bd-ids's unsafe-BD-reuse warning.
Definition at line 253 of file DmaQueueModel.h.
Referenced by guardSequenceQueueDepth().
| LogicalResult xilinx::AIEX::emitUpdateBdAddressFromOffsetParameter | ( | OpBuilder & | builder, |
| Operation * | bdOp, | ||
| BaseMemRefType | bufType, | ||
| uint64_t | registerAddr, | ||
| const AIE::AIETargetModel & | targetModel, | ||
| int | col, | ||
| int | row, | ||
| std::optional< uint64_t > | staticByteAddress = std::nullopt, | ||
| OpFoldResult | lengthField = {} ) |
Definition at line 214 of file AIEUtils.cpp.
References xilinx::AIE::DmaBdLayout::bufferLength, xilinx::AIE::DmaBdLayout::bufferOffset, getBdAddressUnit(), xilinx::AIE::AIETargetModel::getDmaBdLayout(), xilinx::AIE::DmaBdField::place(), xilinx::AIE::DmaBdField::shift, and xilinx::AIE::DmaBdField::word.
Referenced by AIEDMATasksToNPUPass::setAddressForSingleBD().
| LogicalResult xilinx::AIEX::emitUpdateBdLengthFromParameter | ( | OpBuilder & | builder, |
| Operation * | bdOp, | ||
| BaseMemRefType | bufType, | ||
| int64_t | lengthUnit, | ||
| const AIE::AIETargetModel & | targetModel, | ||
| int | col, | ||
| int | row, | ||
| int | bdId ) |
Definition at line 296 of file AIEUtils.cpp.
References xilinx::AIE::DmaBdLayout::bufferLength, xilinx::AIE::AIETargetModel::getDmaBdAddress(), xilinx::AIE::AIETargetModel::getDmaBdLayout(), xilinx::AIE::getLengthUnitBytes(), xilinx::AIE::verifyLengthParameterTile(), and xilinx::AIE::DmaBdField::word.
Referenced by AIEDMATasksToNPUPass::rewriteSingleBD().
| void xilinx::AIEX::encodeHardwareStridesWraps | ( | Policy & | p, |
| uint64_t | elemWidth, | ||
| uint32_t | addressGranularity, | ||
| typename Policy::V | inputSizes[4], | ||
| typename Policy::V | inputStrides[4], | ||
| typename Policy::V | sizes[4], | ||
| typename Policy::V | strides[4] ) |
Definition at line 145 of file BdLowering.h.
Referenced by getHardwareStridesWraps().
| void xilinx::AIEX::eraseDeadArith | ( | mlir::Operation * | root | ) |
Referenced by AIEDMATasksToNPUPass::runOnOperation().
| void xilinx::AIEX::eraseDeadArith | ( | Operation * | root | ) |
Definition at line 87 of file BdLowering.cpp.
| Operation * xilinx::AIEX::findShimDmaStart | ( | AIE::DeviceOp | device | ) |
Definition at line 2713 of file AIEXDialect.cpp.
| void xilinx::AIEX::forEachRuntimePinnedBdId | ( | AIE::DeviceOp | device, |
| llvm::function_ref< void(Operation *op, Operation *tile, std::optional< int > channel, int32_t id)> | pinned ) |
Definition at line 599 of file AIEUtils.cpp.
References getRuntimeDmaTile().
Referenced by xilinx::AIE::DMAChannelAnalysis::TakenBDs::TakenBDs().
| mlir::Value xilinx::AIEX::getAsI64 | ( | mlir::OpBuilder & | builder, |
| mlir::Location | loc, | ||
| mlir::OpFoldResult | ofr ) |
Referenced by buildArgPlusValue(), and guardWithinHostBuffer().
| Value xilinx::AIEX::getAsI64 | ( | OpBuilder & | builder, |
| Location | loc, | ||
| OpFoldResult | ofr ) |
Definition at line 143 of file BdLowering.cpp.
| mlir::Value xilinx::AIEX::getAsValue | ( | mlir::OpBuilder & | builder, |
| mlir::Location | loc, | ||
| mlir::OpFoldResult | ofr, | ||
| mlir::Type | intType ) |
Referenced by getBdRegisterBase(), and DMAStartTaskOpPattern::matchAndRewrite().
| Value xilinx::AIEX::getAsValue | ( | OpBuilder & | builder, |
| Location | loc, | ||
| OpFoldResult | ofr, | ||
| Type | intType ) |
Definition at line 171 of file BdLowering.cpp.
| uint64_t xilinx::AIEX::getBdAddressUnit | ( | const AIE::AIETargetModel & | targetModel, |
| int | col, | ||
| int | row ) |
Definition at line 207 of file AIEUtils.cpp.
References xilinx::AIE::AIETargetModel::getAddressGenGranularity(), and xilinx::AIE::AIETargetModel::isShimNOCTile().
Referenced by emitUpdateBdAddressFromOffsetParameter(), and AIEDMATasksToNPUPass::setAddressForSingleBD().
| mlir::Value xilinx::AIEX::getBdRegisterBase | ( | mlir::OpBuilder & | builder, |
| mlir::Location | loc, | ||
| const xilinx::AIE::AIETargetModel & | targetModel, | ||
| int | tileCol, | ||
| int | tileRow, | ||
| mlir::OpFoldResult | bdId ) |
| Value xilinx::AIEX::getBdRegisterBase | ( | OpBuilder & | builder, |
| Location | loc, | ||
| const AIE::AIETargetModel & | targetModel, | ||
| int | tileCol, | ||
| int | tileRow, | ||
| OpFoldResult | bdId ) |
Definition at line 329 of file BdLowering.cpp.
References createConstantI32(), getAsValue(), and xilinx::AIE::AIETargetModel::getDmaBdAddress().
| std::optional< uint64_t > xilinx::AIEX::getConstantInt64Operand | ( | mlir::Value | v | ) |
Definition at line 1033 of file AIEXDialect.cpp.
Referenced by getConstantIntOperand().
| std::optional< uint32_t > xilinx::AIEX::getConstantIntOperand | ( | mlir::Value | v | ) |
Definition at line 1026 of file AIEXDialect.cpp.
References getConstantInt64Operand().
Referenced by guardSequenceQueueDepth(), syncChannelKey(), and syncCover().
| void xilinx::AIEX::getHardwareStridesWraps | ( | const AIE::AIETargetModel & | targetModel, |
| mlir::Operation * | op, | ||
| mlir::BaseMemRefType | referencedBufType, | ||
| llvm::SmallVector< int64_t, 4 > | inputSizes, | ||
| llvm::SmallVector< int64_t, 4 > | inputStrides, | ||
| llvm::SmallVector< int64_t, 4 > & | sizes, | ||
| llvm::SmallVector< int64_t, 4 > & | strides ) |
Definition at line 111 of file AIEXDialect.cpp.
References encodeHardwareStridesWraps(), and xilinx::AIE::AIETargetModel::getAddressGenGranularity().
Referenced by AIEDMATasksToNPUPass::rewriteSingleBD().
| std::optional< unsigned > xilinx::AIEX::getHostBufferArgIndex | ( | BlockArgument | arg | ) |
Definition at line 149 of file AIEUtils.cpp.
Referenced by AIEDMATasksToNPUPass::setAddressForSingleBD().
| unsigned xilinx::AIEX::getHostBufferCount | ( | Block & | block | ) |
Definition at line 162 of file AIEUtils.cpp.
| bool xilinx::AIEX::getReachableConfigures | ( | Value | task, |
| SmallVectorImpl< DMAConfigureTaskOp > & | configures ) |
Definition at line 527 of file AIEUtils.cpp.
Referenced by getUniqueReachableConfigure().
| Operation * xilinx::AIEX::getRuntimeDmaTile | ( | Operation * | op, |
| SymbolTable & | symbols ) |
Definition at line 578 of file AIEUtils.cpp.
Referenced by AIEGenerateColumnControlOverlayPass::collectTokenIssuingTiles(), and forEachRuntimePinnedBdId().
| AIEX::DMAConfigureTaskOp xilinx::AIEX::getUniqueReachableConfigure | ( | Value | task | ) |
Definition at line 563 of file AIEUtils.cpp.
References getReachableConfigures().
Referenced by DMAStartTaskOpPattern::matchAndRewrite().
|
inline |
Handle a push on key that would land on a full queue: poll where one can be emitted, warn where it cannot (see insertQueueSpaceWait above for when that is).
Warning rather than failing keeps a default from rejecting designs that build today on a target the user can do nothing about; the note is what keeps it from declining quietly.
Definition at line 317 of file DmaQueueModel.h.
References emitQueueOverflowWarning(), insertQueueSpaceWait(), xilinx::AIEX::DmaQueueModel::noteSpaceGuaranteed(), xilinx::AIEX::DmaQueueModel::outstanding(), queueDiagnosedAttr, and xilinx::AIEX::DmaQueueModel::shouldReport().
Referenced by guardSequenceQueueDepth().
|
inline |
Guard every push in a runtime sequence that can land on a full queue, and (when warnUnsafeBdReuse) warn at every Free effect whose own push is not yet proven complete along some reachable path.
Retain all reachable states at branches and loop exits, including zero iterations. Nested loops are analyzed recursively, not walked once. Queue states are finite because overflowing pushes are capped at the target depth; targets without a queued-task model must be ignored rather than accumulated. Analyze each independent channel separately to avoid a Cartesian product of states across channels.
Definition at line 554 of file DmaQueueModel.h.
References xilinx::AIEX::QueueEffect::Await, xilinx::AIEX::DmaQueueModel::awaitToken(), xilinx::AIEX::QueueEffect::Configure, emitUnsafeBdReuseWarning(), xilinx::AIEX::QueueEffect::Free, getConstantIntOperand(), xilinx::AIE::AIETargetModel::getDmaChannelIdleMask(), xilinx::AIE::AIETargetModel::getDmaStatusAddress(), xilinx::AIE::AIETargetModel::getDmaTaskQueueDepth(), xilinx::AIE::AIETargetModel::getDmaTaskQueueSizeMask(), guardQueueOverflow(), xilinx::AIEX::QueueEffect::Ignore, xilinx::AIEX::QueueEffect::issuesToken, xilinx::AIEX::QueueEffect::key, xilinx::AIEX::QueueEffect::kind, xilinx::AIEX::DmaQueueModel::noteSpaceGuaranteed(), xilinx::AIEX::QueueEffect::Push, xilinx::AIEX::DmaQueueModel::push(), xilinx::AIEX::DmaQueueModel::ChannelState::pushes, queueSpacePoll(), recordChannelVisit(), xilinx::AIEX::DmaQueueModel::setState(), xilinx::AIEX::DmaQueueModel::state(), xilinx::AIEX::QueueEffect::task, xilinx::AIEX::DmaQueueModel::ChannelState::unknownTokens, and xilinx::AIEX::DmaQueueModel::wouldOverflow().
| mlir::LogicalResult xilinx::AIEX::guardWithinHostBuffer | ( | mlir::OpBuilder & | builder, |
| mlir::Location | loc, | ||
| mlir::BaseMemRefType | hostBufferType, | ||
| int64_t | baseByteOffset, | ||
| int64_t | elemWidthBytes, | ||
| llvm::ArrayRef< mlir::OpFoldResult > | offsets, | ||
| llvm::ArrayRef< mlir::OpFoldResult > | offsetStrides, | ||
| llvm::ArrayRef< mlir::OpFoldResult > | sizes, | ||
| llvm::ArrayRef< mlir::OpFoldResult > | strides ) |
Referenced by AIEDMATasksToNPUPass::rewriteSingleBDDynamic().
| LogicalResult xilinx::AIEX::guardWithinHostBuffer | ( | OpBuilder & | builder, |
| Location | loc, | ||
| BaseMemRefType | hostBufferType, | ||
| int64_t | baseByteOffset, | ||
| int64_t | elemWidthBytes, | ||
| ArrayRef< OpFoldResult > | offsets, | ||
| ArrayRef< OpFoldResult > | offsetStrides, | ||
| ArrayRef< OpFoldResult > | sizes, | ||
| ArrayRef< OpFoldResult > | strides ) |
Definition at line 253 of file BdLowering.cpp.
References getAsI64().
|
inline |
Definition at line 290 of file DmaQueueModel.h.
References queueSpacePoll().
Referenced by guardQueueOverflow().
|
inline |
Definition at line 65 of file BdLowering.h.
Referenced by verifyConstBdRealizability(), and verifyStridesWraps().
| bool xilinx::AIEX::isContiguousTransfer | ( | llvm::ArrayRef< int64_t > | sizes, |
| llvm::ArrayRef< int64_t > | strides ) |
Definition at line 382 of file AIEXDialect.cpp.
Referenced by isLinearModeTransfer().
| bool xilinx::AIEX::isDecomposableNdDmaPattern | ( | mlir::Operation * | forOp, |
| mlir::BaseMemRefType | referencedBufType, | ||
| const xilinx::AIE::AIETargetModel & | targetModel, | ||
| int | tileCol, | ||
| int | tileRow, | ||
| llvm::ArrayRef< int64_t > | offsetsInnermostFirst, | ||
| llvm::ArrayRef< int64_t > | sizesInnermostFirst, | ||
| llvm::ArrayRef< int64_t > | stridesInnermostFirst ) |
True when the pattern is fully constant, non-contiguous, currently illegal, and decomposeNdDmaPattern would produce at least one legal replacement.
| bool xilinx::AIEX::isLinearModeTransfer | ( | const AIE::AIETargetModel & | targetModel, |
| int | col, | ||
| int | row, | ||
| llvm::ArrayRef< int64_t > | sizes, | ||
| llvm::ArrayRef< int64_t > | strides ) |
Definition at line 393 of file AIEXDialect.cpp.
References isContiguousTransfer(), isLinearTransfer(), and xilinx::AIE::AIETargetModel::isShimNOCTile().
Referenced by aieDmaBdFits(), and AIEDMATasksToNPUPass::rewriteSingleBD().
| bool xilinx::AIEX::isLinearTransfer | ( | llvm::ArrayRef< int64_t > | sizes, |
| llvm::ArrayRef< int64_t > | strides ) |
Definition at line 370 of file AIEXDialect.cpp.
Referenced by isLinearModeTransfer().
| bool xilinx::AIEX::namesFifo | ( | Operation * | op, |
| StringRef | fifo, | ||
| SymbolTableCollection & | symbols ) |
Definition at line 386 of file AIEUtils.cpp.
| bool xilinx::AIEX::patternPassesVerification | ( | mlir::Operation * | forOp, |
| mlir::BaseMemRefType | referencedBufType, | ||
| const xilinx::AIE::AIETargetModel & | tm, | ||
| int | tileCol, | ||
| int | tileRow, | ||
| const NdDmaPattern & | pattern, | ||
| bool | skipTransformationChecks = false ) |
Returns true when the pattern passes verifyStridesWraps for the given tile and memref element type.
Diagnostics are suppressed so callers can probe legality without emitting errors.
Referenced by aieDmaBdFits().
|
inline |
Address and mask for a poll that waits for a free task-queue slot.
Unavailable when the target reports no pollable occupancy register, or when depth is not a power of two – a maskpoll tests masked equality, not "<", so "occupancy < depth" has to be the single "depth bit is clear" test (for depth 4, bit 22 of Task_Queue_Size).
Definition at line 274 of file DmaQueueModel.h.
References xilinx::AIE::AIETargetModel::getDmaStatusAddress(), and xilinx::AIE::AIETargetModel::getDmaTaskQueueSizeMask().
Referenced by guardSequenceQueueDepth(), and insertQueueSpaceWait().
|
inline |
What a push or an await that names no task does to its channel: an aiex.npu.push_queue; an aiex.npu.dma_memcpy_nd through the placed shim allocation its metadata names, which always issues a token on S2MM (its lowering sets one); an aiex.npu.sync syncChannelKey pins down; an aiex.npu.dma_wait on a placed shim allocation.
Ignore for anything else.
Definition at line 392 of file DmaQueueModel.h.
References xilinx::AIEX::QueueEffect::await(), xilinx::AIEX::DmaQueueModel::keyOf(), xilinx::AIEX::QueueEffect::push(), and syncChannelKey().
|
inline |
Record op and each operation enclosing it up to body's own op into blocks, once per block: an ancestor chain's visited block repeats until the chain's op there changes.
Definition at line 532 of file DmaQueueModel.h.
Referenced by checkTokensConsumed(), and guardSequenceQueueDepth().
| void xilinx::AIEX::registerAIEResourceStagesPipeline | ( | ) |
Definition at line 102 of file AIEResourceStages.cpp.
References xilinx::AIEX::AIEResourceStagesOptions::adaptiveOverlay, addBufferAddressStage(), addLoweringStage(), addPlacementStage(), addRoutingStage(), addTraceStage(), xilinx::AIEX::AIEResourceStagesOptions::allowDeadlockProneRouting, xilinx::AIEX::AIEResourceStagesOptions::coresPerCol, xilinx::AIEX::AIEResourceStagesOptions::ctrlPktOverlay, xilinx::AIEX::AIEResourceStagesOptions::dynamicObjFifos, xilinx::AIEX::AIEResourceStagesOptions::loadPdiToCtrlPkt, xilinx::AIEX::AIEResourceStagesOptions::packetSwObjFifos, xilinx::AIEX::AIEResourceStagesOptions::placementBudget, xilinx::AIEX::AIEResourceStagesOptions::placer, xilinx::AIEX::AIEResourceStagesOptions::saEffort, xilinx::AIEX::AIEResourceStagesOptions::saSeed, and xilinx::AIEX::AIEResourceStagesOptions::skipObjectFifoVerify.
Referenced by aieRegisterAllPasses().
| std::string xilinx::AIEX::scratchpadCommandName | ( | AIE::RuntimeSequenceOp | seq | ) |
Definition at line 451 of file AIEUtils.cpp.
| std::optional< int64_t > xilinx::AIEX::staticExecutions | ( | Operation * | op | ) |
Definition at line 329 of file AIEUtils.cpp.
|
inline |
The channel an aiex.npu.sync retires on, or nullopt when it cannot be pinned down.
Its six operands are exactly a ChannelKey, with direction numbered as AIE::DMAChannelDir because DmaWaitToSyncPattern builds that operand by casting the enum. Unresolvable retires nothing, which over-reports; guessing the other way would hide a real hang.
Definition at line 164 of file DmaQueueModel.h.
References getConstantIntOperand().
Referenced by awaitSync(), checkTokensConsumed(), and rawQueueEffect().
|
inline |
Definition at line 196 of file DmaQueueModel.h.
References xilinx::AIEX::SyncCover::chan, xilinx::AIEX::SyncCover::colHi, xilinx::AIEX::SyncCover::colLo, xilinx::AIEX::SyncCover::dir, getConstantIntOperand(), xilinx::AIEX::SyncCover::rowHi, and xilinx::AIEX::SyncCover::rowLo.
Referenced by checkTokensConsumed().
| std::optional< AIEX::SubviewTraceResult > xilinx::AIEX::traceSubviewToBlockArgument | ( | Value | value | ) |
Definition at line 25 of file AIEUtils.cpp.
Referenced by AIEDMATasksToNPUPass::rewriteSingleBDDynamic(), and AIEDMATasksToNPUPass::setAddressForSingleBD().
|
inline |
Definition at line 77 of file BdLowering.h.
References isConstMultipleOfGranule().
Referenced by AIEDMATasksToNPUPass::rewriteSingleBD().
| mlir::LogicalResult xilinx::AIEX::verifyStridesWraps | ( | mlir::Operation * | forOp, |
| llvm::function_ref< mlir::InFlightDiagnostic()> | emitError, | ||
| mlir::BaseMemRefType | referencedBufType, | ||
| int | tileCol, | ||
| int | tileRow, | ||
| llvm::SmallVector< int64_t, 4 > | inputSizes, | ||
| llvm::SmallVector< int64_t, 4 > | inputStrides, | ||
| llvm::SmallVector< int64_t, 4 > | hardwareSizes, | ||
| llvm::SmallVector< int64_t, 4 > | hardwareStrides, | ||
| bool | skipTransformationChecks = false ) |
Definition at line 150 of file AIEXDialect.cpp.
References xilinx::AIE::getTargetModel(), and isConstMultipleOfGranule().
Referenced by AIEDMATasksToNPUPass::rewriteSingleBD().
| bool xilinx::AIEX::wholeGranules | ( | const Volume & | bytes, |
| int64_t | granule ) |
Definition at line 371 of file AIEUtils.cpp.
Referenced by balanceUnder().
|
inlineconstexpr |
Name of the runtime sequence aie-expand-load-pdi{init-entry} adds.
Definition at line 100 of file AIEXPasses.h.
|
inlineconstexpr |
Definition at line 309 of file DmaQueueModel.h.
Referenced by guardQueueOverflow().