ztachip Hardware Architecture

1.2 Subcomponents

1.3 Functions

This is the top-level component of ztachip.

The central tensor processor unit is dp_core

dp_core receives tensor instructions from RISCV via axilite_* interface.

dp_core then executes the tensor instructions by performing the following:

• Coordinating tensor data operations which transfer tensor data between sram_core,

core's internal memory and external DDR memory. Tensor data operations may also include other complex functions such as transpose, dimension resize, data-remap...

• Dispatching tensor operator execution requests to core which then in turn dispatch the execution to an array of VLIW processors.
• Dispatching tensor floating point and aggregate arithmetic executions to fpu

2. ztachip.dp_core

2.1 Interfaces

2.2 Subcomponents

2.3 Functions

This component is the main tensor processor of ztachip. It performs the following functions:

• dp_fetch receives tensor instructions from RISCV. Each tensor instruction are associated

with a hardware thread. There are 2 hardware threads available. Hardware threads are useful to provide the ability to overlay the tensor operator execution phase of one thread with the data transfer phase of the other threads.

• Decodes the tensor instructions.
• Tensor instructions may be executed out-of-order but applications can enforce the order.
• Data operations are then processed by dp_gen_core,

dp_source and dp_sink. There can be up to 2 data operations executing at the same time. For example, data transfer from the core's internal memory to DDR external memory can occur at the same time as data transfer from the scratch-pad to core's internal memory.

• Tensor operator execution is forwarded to core via interface signal task*
• Before tensor operator execution can be issued, all memory transfer with core's internal memory must be completed. Since there is a separate core's internal memory for each hardware thread, for example, memory transfer to core's internal memory belonging to thread#1 can still be running at the same time while core is busy performing tensor operator execution but on thread#2.

3. ztachip.fpu

3.1 Interfaces

3.2 Subcomponents

3.3 Functions

FPU supports floating point arithmetic with different precision below

• FP32: Many transformer intermediate calculations should stay in FP32
• BFLOAT: Transformer activations are normally stored in BFLOAT

Aggregate functions are also better supported by FPU. Aggregate functions are inherently serial operations and better supported with FPU's data flow architecture instead. The following aggregate functions are supported

• MAX
• SUM
• DOT-PRODUCT-SUM

FPU operates in vector mode. FPU input/output are vectors of float numbers. FPU vector width is configured in config.vhd as fpu_gen_max_c.

FPU execution is fully pipelined. For example, for the all important FMA operations found in LLM attention stage, it can execute up to 2*fpu_gen_max_c float operations (FMA=1ADD+1MUL) per clock cycle.

Data fetching is streamed to FPU from SRAM. Data fetching are overlapped with execution, therefore adding no delay to FPU execution.

FPU operates from SRAM memory space only

• FPU data input must first be transferred from PCORE or DDR space to SRAM space by DP instructions
• FPU results must be transferred to PCORE or DDR space from SRAM space by DP instructions

For highest performance, FPU should be used under both HART context. There is 1 FPU hardware instance but it can operate under different HART context. One can use one HART context to transfer data between SRAM and DDR/PCORE while the other HART context is still busy executing FPU instructions. This way, all memory overhead are overlapped with FPU execution.

4. ztachip.core

4.1 Interfaces

4.2 Subcomponents

4.3 Functions

• This component is responsible for all tensor operator execution tasks.
• It is composed of an array of pcores. And each pcore is a VLIW vector processor.
• Requests for tensor operator execution are coming from dp_core via task* interface signal.
• The component instr is controlling the execution of all pcore's VLIW processors.

VLIW processors are very lightweight processors that are mostly just ALU with memory running in lock-step mode with each other.

5. ztachip.core.stream

This module provides an arbitrary data mapping between input value and output.

stream performs data mapping on data just before it is written to pcore's memory space.

stream also performs data mapping on data just after it is read from pcore's memory space.

The data mapping is realized using the formula below

y = lookup_quotient[x[11:5]] + x[4:0]* lookup_remainder[x[4:0]]

Effectively, the formula above provides a multi-point linear-interpolation between input and output.

stream provides up to 4 different data-mapping functions at the same time. The parameter stream_id_in selects which data mapping to be performed.

stream is very commonly used in AI applications to perform activation functions as data being written back to memory.

6. ztachip.core.pcore

6.1 Interfaces

6.2 Subcomponents

6.3 Functions

This component performs tensor operator execution.

There are many instances of pcore running in parallel.

pcore has a VLIW architecture.

pcore execution is multi-threaded. There are 16 hardware threads of execution running in round-robin fashion.

The VLIW instruction is very wide that contains many sub-functions that perform the following tasks simultaneously within a single VLIW instruction.

• Address calculation for 2 input parameters and 1 output parameter.
• Fetching input parameter vectors from register_bank.
• Fetching 32-bit accumulator from xregister_file. These accumulator values are then used by alu performing FMA (Fused-multiply-add) operations.
• Performing integer calculations on ialu.
• Perform vector calculation on the bank of alu.
• alu may produce results as 32-bit accumulator, save these values to xregister_file.
• alu may produce results as 12-bit word vector. Save these results back to

register_bank.

Since all the functions above are simultaneously performed in one single VLIW instruction, the instruction pipeline is as long as 14 clocks. However, with hardware multi-threading execution, each stage of the execution pipeline is occupied by different threads. Therefore we have an effective execution rate of one VLIW instruction per clock per pcore.

For example, the code below is compiled into one single VLIW instruction. All data operations and arithmetic calculation together take just one clock of execution and also without any memory stall cycles. This same code using traditional RISCV instructions would take up to 10 RISCV instructions that may also incur memory stall cycles in addition.

z[i++] = x[i+2]+y[i+3];

7. ztachip.core.pcore.register_bank

This component implements pcore internal memory

pcore's internal memory is partitioned into 2 pages with each page assigned to one of the two tensor processor's threads. This is an important concept for ztachip since it allows for one thread to access its internal memory while the other thread is performing tensor operator execution using the other internal memory's page. This allows for a memory operation to be overlapped with tensor operator execution.

It is important to note that tensor operator execution operates only on internal memory without any reference to external memory. This is an important concept since with ztachip, data operations are decoupled from execution operations by having separate tensor instructions for data operations and execution operations. Reference here for further explanation of this concept.

Internal memory holds 2 types of data

• Private data: Data is private to each of the 16 threads. Private memory words are interleaved between different threads as shown in the picture below.
• Shared data: Data is shared among all 16 threads but within the same pcore. Shared memory words are allocated from the bottom up as shown in the picture below.

+-------------------------+
+ Thread1.Private word#0  |
+-------------------------+
+ Thread2.Private word#0  |
+-------------------------+
           :
           :
+-------------------------+
+ Thread16.Private word#0 |
+-------------------------+
+ Thread1.Private word#1  |
+-------------------------+
+ Thread2.Private word#1  |
+-------------------------+
           :
           :
+-------------------------+
+ Thread16.Private word#1 |
+-------------------------+
           :
           :
+-------------------------+
+ Shared           word#3 |
+-------------------------+
+ Shared           word#2 |
+-------------------------+
+ Shared           word#1 |
+-------------------------+
+ Shared           word#0 |
+-------------------------+

8. ztachip.core.pcore.instr_decoder2

This is the main component that decodes the computing components of VLIW instructions

There are 3 main operations to each VLIW instruction:

• operation on pcore.alu: To perform math operations on vector data
• operation on pcore.ialu: To perform simple scalar integer operation.
• branching operation. But this is performed by master processor core.instr since

all VLIW processors are running in lockstep therefore sharing the same branching operation. ztachip currently does not support yet thread branching/divergence as commonly implemented in GPU, therefore all threads must have the same branching condition. However you can still perform the equivalent of branching with conditional execution using VMASK register as described in next section. Most AI/Vision processing tasks however are mostly predictable computing with minimum or no branching.

VLIW instructions are 128 bit long and have the following format

MU                 -- pcore.alu instruction
   oc         5    -- opcode
   save       1    -- Save alu.y_out=>xregister_file
   x3              -- x3 parameter;map to alu.xscalar_in
      vector  1    -- 1 if parameter is vector,0 if scalar
      addr    12   -- Internal memory address of x3
      attr    4    -- Attribute
   x1              -- x1 parameter;map to alu.x1_in
      vector  1    -- 1 if parameter is vector,0 if scalar
      addr    12   -- Internal memory address of x1
      attr    4    -- Attribute
   x2              -- x2 parameter;map to alu.x2_in
      vector  1    -- 1 if parameter is vector,0 if scalar
      addr    12   -- Internal memory address of x2
      attr    4    -- Attribute
   y               -- y parameter;map to alu.y*_out
      vector  1    -- 1 if parameter is vector,0 if scalar
      addr    12   -- Internal memory address of y
      attr    4    -- Attribute
IMU                -- pcore.ialu instruction
   oc         5    -- opcode
   x1         4    -- x1 parameter; map to ialu.x1_in
   x2         4    -- x2 parameter; map to ialu.x2_in
   y          4    -- y parameter; map to ialu.y_out
   const      13   -- Constant field
CTRL
   oc         5    -- Branching opcode based on IMU.y value
   addr       11   -- Code address to jump to

MU.attr field above provides many ways for parameters to reference memory.

Table below shows different addressing options for parameters.

where IREG is ialu's register file and MEM is pcore's internal memory.

ATTR  DESCRIPTION                  MEMORY ACCESS PATTERN
===========================================================
11RR  Access via pointer+index     MEM[IREG[addr]+IREG[RR]]
1011  Access via pointer           MEM[IREG[addr]]
1000  Access shared memory         MEM[addr]
1001  Access private memory        MEM[addr]
00RR  Access shared memory+index   MEM[addr+IREG[RR]]
01RR  Access private memory+index  MEM[addr+IREG[RR]]

9. ztachip.core.pcore.ialu

This component is responsible for the scalar integer calculation of VLIW instruction described above.

ialu operates using the following registers

• R0-R7 : General purpose integer registers.
• VMASK : Register with each bit is used to enable/disable a vector lane.

The integer calculations provided by ialu are typically used for tasks such as

• Loop counter/control
• Index to an array
• Memory reference by pointer
• Address calculation
• Enable/disable vector lanes.

For example

// i is an integer operation used for loop counter
for(i=0;i < 10;i++) {
   :
   :
}

// i is an integer used for array indexing and address calculation
z[i]=x[i]+y[i];

// p is an integer used for pointer reference to memory
// Both integers p and i are used for memory reference and
// address calculation below
p=&mem[0];
p[i]=10;

// VMASK=3 means only element#0 and element#1 of vector z are
// being updated after the vector calculation
VMASK=3;
z=x+2; // Only the first 2 elements of z are written back to memory.

10. ztachip.core.pcore.alu

10.1 Interfaces

10.2 Functions

ALU has a fairly simple set of operations. We want the ALU to be as simple as possible to reduce resource requirements. They are meant to perform linear arithmetic operations occurring normally during AI and vision processing tasks.

Other non-linear operations are taken care of by stream.

ALU and stream together were shown to be able to cover a wide range of AI and vision processing.

ALU has the following arithmetic blocks:

• Multiplication of two 12 bit values.
• Adding the result of multiplication to a 32-bit accumulator value.
• Perform shift operation
• Perform boolean comparison of the result. Comparison against zero include GT,GE,LT,LE,EQ,NE
• Casting the 32-bit result to 12 bit result. Value is clipped if it is overflowed or underflowed after casting.

With the above blocks, the following opcodes are supported

  COMPARE_LT       : y2_out=(y_out < 0)?1:0
  COMPARE_LE       : y2_out=(y_out <= 0)?1:0
  COMPARE_GT       : y2_out=(y_out > 0)?1:0
  COMPARE_GE       : y2_out=(y_out >= 0)?1:0
  COMPARE_EQ       : y2_out=(y_out==0)?1:0
  COMPARE_NE       : y2_out=(y_out!=0)?1:0
  MULTIPLY         : y_out=x1*x2
  ADD              : y_out=x1+x2
  SUBTRACT         : y_out=x1-x2
  FMA              : xreg_in += x1_in*x2_in
  FMS              : xreg_in -= x1_in*x2_in
  ASSIGN           : y_out=x1_in
  ACCUMULATOR_SHL  : y_out=(xreg_in << x_scalar_in)
  ACCUMULATOR_SHR  : y_out=(xreg_in >> x_scalar_in)
  INT12 SHR        : y_out=(x1_in >> x_scalar_in)
  INT12 SHL        : y_out=(x1_in << x_scalar_in)