Belle II DAQ

Size: px

Start display at page:

Download "Belle II DAQ"

そよこやぎ
6 years ago
Views:

1 Belle II 実験データ収集システム山田悟 (KEK 素核研 ) 函館アリーナ

2 1. イントロダクション 2. Belle II データ読み出しシステム 3. DAQシステムのパフォーマンス 4. 外層検出器の宇宙線試験 5. Belle II 読み出し系のアップグレード計画函館アリーナ

SuperKEKB accelerator Designed luminosity: 40times as large as KEKB 50 ab -1 in 10 years (cf.

3 Belle II 実験 Search for new physics beyond the Standard Model(SM) via high precision measurement with high statistics samples of B/D/tau decays. SuperKEKB accelerator Designed luminosity: 40times as large as KEKB 50 ab -1 in 10 years (cf. 1ab Belle experiment) Belle II collaboration : ~750 collaborators from 24 countries Increase of beam current Smaller beam size x2 x20 luminosity x40 Improved 函館アリーナ

4 We are here SuperKEKB のルミノシティ ( 予測 ) Phase I : (2016 Feb.-Jun.) Accelerator commissioning w/o final focusing magnets w/o the Belle II detector First turns of SuperKEKB Vacuum scrubbing Phase II : (2018Feb.-Jul) Accelerator commissioning and physics run with the Belle II detector except for vertex subdetectors Phase III : (around the end of 2018-) Physics run with the full Belle II detector 函館アリーナ

各 detector のインストール状況 ECL( 電磁カロリメータ ) -> install 済み崩壊点検出器 (PXD,SVD) 完全な install は 2018 年 KLM(Klong muon 検出器 ) -> install 済み ARICH( エアロジェルリングイメージングチェレンコフカウンタ

5 各 detector のインストール状況 ECL( 電磁カロリメータ ) -> install 済み崩壊点検出器 (PXD,SVD) 完全な install は 2018 年 KLM(Klong muon 検出器 ) -> install 済み ARICH( エアロジェルリングイメージングチェレンコフカウンタ ) 実験ホールで組立作業 2017 年に install CDC( 中央飛跡検出器 ) 実験ホールに仮設置 10 月に Belle II 検出器に install TOP(Time of Propagation) カウンタ -> install 済み函館アリーナ

6 測定器設置コミッショニング等のスケジュール現在 Jan. Feb. Mar. Apr. May Jun. Jul. Aug. Sep. Oct. Nov. Dec. Jan. Feb. Mar. PXD SVD 測定器作製組み立て作業測定器組み立て作業 Phase II 用測定器組み立て作業測定器設置測定器の一部を組み込む CDC TOP ARICH ECL KLM 測定器設置測定器設置測定器設置 (barrel 部 ) 測定器設置 Belle II 測定器ロールイン磁場中統合宇宙線試験測定器組み立て試験磁場中統合宇宙線試験測定器設置統合 / 独立での宇宙線試験など Phase II beam run (2018 July まで ) 函館アリーナ

7 1. イントロダクション 2. Belle II データ読み出しシステム 3. DAQシステムのパフォーマンス 4. 外層検出器の宇宙線試験 5. Belle II 読み出し系のアップグレード計画函館アリーナ

Belle II データ収集システム ( 今回は読み出しシステムの話を主に ) - FEE との interface は各検出器共通 (PXD 以外 ) - HLT による rate reduction + Region of Interest による PXD の event size reduction Trigger and Timing distribution Level1

8 Belle II データ収集システム ( 今回は読み出しシステムの話を主に ) - FEE との interface は各検出器共通 (PXD 以外 ) - HLT による rate reduction + Region of Interest による PXD の event size reduction Trigger and Timing distribution Level1 trigger ~ 30kHz (max. value for DAQ development ) Data-reduction w/ ROI for PXD 内層 ( 崩壊点 ) 検出器 (Phase III から ) Common readout-system for sub-detectors Event-building Event-building And storage 外層検出器 (phase II から ) Software Eventreduction X210 readout boards X40 Readout PCs 1GbE/10GbE x10 High Level Trigger+storage unit switch 函館アリーナ HLT: ( 20nodes x 16cores )/ unit 8

9 フロントエンド検出器とバックエンド DAQ との接続 2 つのインターフェイスが必要 FEE とトリガータイミング分配システム : FEE とバックエンド DAQ ( データフロー ) データフローについては各検出器に共通の通信用 firmware (Rocket I/O ベース ) を使用 belle2link FEE board Trigger/clock Backend DAQ Backend-DAQ (DAQ gr.) FIFO FPGA on Front-end electronics board 函館アリーナ

フロントエンド電子回路からのデータ読み出し Belle2Link : (D. Sun et. all, hysics Procedia Volume 37, 2012, pp.

signal with data transmission based on Rocket I/O FEE side : Functions for I/F with FEE and Trigger

electronics A/D conver sion FPGA data FEE I/F GTP link Belle2Link Line rate : 3.

10 フロントエンド電子回路からのデータ読み出し Belle2Link : (D. Sun et. all, hysics Procedia Volume 37, 2012, pp ) Unified high speed link which connects Front-End Electronics (FEE) and DAQ system for signal with data transmission based on Rocket I/O FEE side : Functions for I/F with FEE and Trigger Timing Distribution on FPGA DAQ side : High Speed Link Board() as a data receiver Front-end electronics A/D conver sion FPGA data FEE I/F GTP link Belle2Link Line rate : 3.125Gbps data configutaion by register access (Virtex5) : data readout board I/F data board Developed by IHEP 函館アリーナ

new daughter-boards for Belle II are used PrPMC on CPU: Intel Atom 1.

11 読み出しボードでのデータ処理 Readout board : ( COmmon Pipelined. Platform for Electronics Readout ) Versatile DAQ board developed at KEK -> basically same functionality in the previous Belle experiment can be equipped with various I/O cards and CPU card -> new daughter-boards for Belle II are used PrPMC on CPU: Intel Atom 1.6GHz Z530P DDR2 SDRAM 512MB PXE boot from ROPC Gigabit Ethernet x1 board PMC processor Onboard FIFO Device Driver CPU (PrPMC) Read Data Process data Send data Ethernet Readout PC Belle2link Trigger/clock GbE port x2(onboard and PrPMC) Data processing on CPU Data formatting (Add header and trailer to raw data) Plain data check Event incrementation, check magic word etc. Add XOR checksum Report data-flow status to slow control 函館アリーナ

12 読み出し PC 上でのデータ処理 I. data check by data-handler process I. Calculate CRC16 and compare CRC value attached by FEE II. XOR checksum calculated by software on II. Data size reduction merging redundant header/trailer attached by b2link and ) Reduction by 15MB/s/ROPC at 30kHz trigger rate( <- 5s/ROPC, 4/ ) III. Collect data from several s and do partial event-building and send data to High level trigger unit. ROPC( Readout PC) Gigabit Ethernet Network switch Gigabit Ethernet Data handler Data handler Data handler Partial Event builder Gigabit Ethernet Network switch 10GbE Event builder/ High level trigger Other ROPCs 函館アリーナ

13 1. イントロダクション 2. Belle II データ読み出しシステム 3. DAQシステムのパフォーマンス 4. 外層検出器の宇宙線試験 5. Belle II 読み出し系のアップグレード計画函館アリーナ

各検出器での Belle II のイベントサイズ見積もり More detailed data size estimation for some

1input / 4inputs / MC result (ROOT objects) Add header/footer, Fill data

4inputs / Difference of event size is handled by the number of receiver

14 各検出器での Belle II のイベントサイズ見積もり More detailed data size estimation for some sub-detectors with MC data to consider assignment of readout boards. 1input / 4inputs / MC result (ROOT objects) Add header/footer, Fill data in raw-data format Packer software Raw data SVD CDC TOP 4inputs / ARICH 4inputs / Difference of event size is handled by the number of receiver cards on SVD : 1s/ ECL : 2s/ CDC/TOP/ARICH/KLM : 4s/ 函館アリーナ

15 パフォーマンス測定 (1): FEE と CPU Detec tor FEE COPP ER ROPC HLT/st orage Test setup Tested here Dummy trigger source Readout PC CDC FEE CDC FEE CDC FEE CDC FEE Belle2link Belle2link PrPMC nc > /dev/null We can test data-transfer performance of belle 2link CPU usage on PrPMC 函館アリーナ

16 結果 : CPU Input usage trigger on rate = 30kHz PrPMC Throughput from Input trigger rate = 30kHz SVD ECL others 30kHz operation was achieved. CPU usage will be the bottleneck when the event size becomes larger than expected. Throughput in Belle2link and Gigabit Ethernet to a readout PC has still enough remaining room. CDC TOP ARICH KLM PrPMC PrPMC PrPMC ECL SVD 函館アリーナ

17 パフォーマンス測定 (2) : -> readout PC 1ROPC and several s. # of s differs over sub-detectors due to the difference of event size Provide trigger to board to produce dummy data by. Detec tor FEE COPP ER Tested here ROPC HLT/st orage Trigger source Trigger ROPC( Readout PC) Intel(R) Xeon(R) CPU E v2 2.60GHz High level trigger server dumhslb dumhslb dumhslb dumhslb dumhslb dumhslb dumhslb dumhslb dumhslb dumhslb dumhslb dumhslb dumhslb dumhslb dumhslb dumhslb dumhslb dumhslb dumhslb PrPMC dumhslb dumhslb dumhslb dumhslb dumhslb dumhslb dumhslb dumhslb dumhslb Data handler Data handler Data handler Partial Event builder nc > /dev/null Data source : Use FPGA as a dummy-data producer We can test Processing power of and ROPC data-transfer performance between and ROPC, ROPC and HLTin. 函館アリーナ

-> readout PC 結果 : Throughput on ROPC CPU usage on Trigger rate = 30kHz 35kHz for SVD is the max. event rate. Bottleneck : Output data flow to HLT is near the limit of GbE.

18 -> readout PC 結果 : Throughput on ROPC CPU usage on Trigger rate = 30kHz 35kHz for SVD is the max. event rate. Bottleneck : Output data flow to HLT is near the limit of GbE. CPU usage on CPU is still room to increase the rate Increase # of Readout PCs or increase throguhtoput between ROPC and HLT will increase the limit. 函館アリーナ

19 ECL 検出器での high rate test の結果 - Setup : 36FEE -> 18 s -> 7 readout PC - Throughput : about 33MB/s/ ( the expected event size from MC ) - Event size was adjusted by HIT threshold of ECL FEE - Total throughput for Barrel ECL : 600 MB/s - Constant 30kHz trigger -> efficiency = 99.2% - Pseudo-Poisson 30kH trigger -> efficiency = 98.2 % - The deadtime comes from trigger limitation (5trigger in 26us due to SVD FEE ). Constant 30kHz trigger rate Pseudo-Poisson 30kHz trigger rate Efficiency = 99.2% Efficiency = 98.4% CPU usage on a board -> High efficiency(nearly 100%) was achieved! 函館アリーナ

20 ボード上でのデータ化け問題 1. ff00ff00 error ボード ( 上のデータ受信用ドーターカード ) の Virtex5 FPGA Inside hslb_receiver.vhd State machine 1 Receve data from GTP and store them to FIFO_rx FIFO_rx State machine 2 Read data fromfifo_rx and store them to FIFO FIFO 2. Bit error 函館アリーナ

21 How CRC error is detected Front-end electronics board Readout PC FEE ROPC DAQ software checks the checksums Raw XOR driver XOR b2link packet CRC (CRC value per packet) b2link core Error count is stored in data b2link event CRC (CRC value per event)? Driver DAQ software b2link packet CRC has not been observed. -> b2link transmission is O.K. But b2link event CRC error was detected. -> Data should be corrupted after receiving data from a FEE. Stored in data Not stored, because most of header is removed. CRC info. is stored in data. -> re-calculated from data by readout PC 函館アリーナ CRC errors

22 1. FPGA 内でのデータ化け : header/footer attached by - A large amount of ff00ff00 appeared after an FEE footer - b2link packetcrc error is not detected. -> data corruption after received data. : header/footer attached by FEE : data contents of FEE : strange data Data of slotd (corrupted data) ffaa41b5 ff000b4d b8c b55881 f7af0004 d4000b4d c8c f b50b4d b8c741b5 7b36fe00 ff00ff00 ff00ff00 ff00ff00 ff00ff00 ff00ff00 ff00ff00 ff00ff00 ff00ff00 ff00ff00 ff00ff00 ff00ff00 ff00ff00... ff00ff00 ff00ff00 ff00ff00 ff00ff00 ff00ff00 ff00ff00 ff00ff00 ff00ff00 ff00ff00 ff00ff00 ff Inside hslb_receiver.vhd Belle II : 1 ワード = 32 ビット (4 バイト ) State machine 1 Receve data from GTP and store them to FIFO_rx FIFO_rx State machine 2 Read data fromfifo_rx and store them to FIFO FIFO Data check CRC32 check -> O.K. CRC16 check -> N.G. -> Data should be corrupted around FIFO in FPGA 函館アリーナ

23 Why data are filled with ff00ff00 Data of slotd (corrupted data) ffaa41b5 ff000b4d B8c b55881 b8c741b5 7b36fe00 Ff00ff00 ff00ff00 When FIFO is empty, the output is ff00. For some reason, ff00 is inserted at the beginning of the event. Data are shifted by 2bytes. fe00 is the delimiter to indicate the end of the event. But due to the 2byte shift, this delimiter is ignored and empty FIFO is read repeatedly, which returns ff00. Workaround to avoid the first ff00 Just ignore if the 1 st byte of an event from FIFO_rx is ff. The 1 st byte is supposed to never be ff. 函館アリーナ

24 2. ボード上の bit エラー Data corruption in ffffffff pattern Effect of SSO (simultaneous switching outputs )? A. どのように化けるか [DEBUG] ffffffff ffffffff ffffffff ffffffff ffffff [DEBUG] ffffffff ffffffff ffffffff ffffffff ffffffff [DEBUG] ffffffff ffffffff ffffffff ffffffff ffffffff [DEBUG] ffffffff ffffffff ffffffff effffff ffffffff B. Reduction of the current drive of data output works : in hslb_***.ucf. (default 12mA to 2mA) Errors after the modification at the B2/B3 test bench B3 setup 12x (4/) Input trigger 30kHz Poisson : output trigger 1.1kHz Data pattern : ffffffff No data corruption in 118.5hours for 323.3Mevents しかしこれでもまだ TOP 検出器の CRC エラーは解決せず ( 次ページ ) 函館アリーナ

25 Data corruption in feffffff pattern A. TOP 検出器データの化け方 Using the output log of an error event, I put the same data pattern to dumhslb firmware. Data corruption occurred in the B3 test bench and the data pattern seemed to be similar in error events. The red bits became 0 in the corrupted events. feff0400 fefffdff feff ffff fcfff9ff f5fff7ff f5fffbff feff0400 fefffdff feff ffff fcfff9ff f5fff7ff f5fffbff fefffbff f6fff6ff ffff f9fffdff fafffeff f7fff6ff B. テストパターン feffffff We tried feffffff pattern and it caused data corruption. [DEBUG] feffffff feffffff feffffff feffffff feffffff [DEBUG] feffffff feffffff feffffff feffffff feffffff fbffffff also caused data corruption fbffffff fbffffff fbffffff fbffffff fbffffff On the other hand, no errors in 2hours with fffeffff 函館アリーナ

C. オシロで feffffff 00000000 を探す FF lines(0 31) driver FIFO1 FIFO2 oscilloscope We soldered probe lines on a board but no data corruption was detected by an oscilloscope before the 1st FIFO. D.

26 C. オシロで feffffff を探す FF lines(0 31) driver FIFO1 FIFO2 oscilloscope We soldered probe lines on a board but no data corruption was detected by an oscilloscope before the 1st FIFO. D. ボードの個体差もあるようである # of CRC errors in feffffff test pattern So far, no prospect of fixing this problem. Since the error rate differs in () boards, we are considering replacing some TOP s to reduce the error rate. 函館アリーナ

27 1. イントロダクション 2. Belle II データ読み出しシステム 3. DAQシステムのパフォーマンス 4. 外層検出器の宇宙線試験 5. Belle II 読み出し系のアップグレード計画函館アリーナ

統合宇宙線テスト 2017 年 7 月,8 月 : QCS( 収束磁石 ), Belle II ソレノイド

ARICH については現在開発および試験段階なので参加せずトリガー : CDC track segment

finder ロジックを使用 Trigger rate Back-to back ( 同色の 2 つの

TSF && ECL(timing) : ~100Hz 宇宙線テストでの event rate CDC

28 統合宇宙線テスト 2017 年 7 月,8 月 : QCS( 収束磁石 ), Belle II ソレノイド (1.5T) を定格運転した状態で宇宙線測定測定器 :CDC TOP ECL KLM PXD, SVD, ARICH については現在開発および試験段階なので参加せずトリガー : CDC track segment finder + ECL timing 1 super-layer の track segment finder ロジックを使用 Trigger rate Back-to back ( 同色の 2 つの segment を通ることを要求 ) TSF && ECL(timing) : ~10Hz Single TSF && ECL(timing) : ~100Hz 宇宙線テストでの event rate CDC をビーム方向から見た図色がついているのが今回使用した trigger の segment 函館アリーナ

Decigeon Logic Trigger/Timing distribution network Run-control FEE Data flow Serial link Readout board Ether net

29 宇宙線試験でのデータ収集システム実際のビームランで使う DAQ システムを使用 Front-End Electronics boards はそれぞれの測定器で異なる FEE -> 読み出しボードの protocol は統一されており backend DAQ は各測定器共通 BelleII Detector Electronics Hut Server room Slow control Trigger subsystem Global Decigeon Logic Trigger/Timing distribution network Run-control FEE Data flow Serial link Readout board Ether net Readout Readout Readout PC PC PC Ether net HLT/storage server Stora ge 129 s 21 readout PCs 函館アリーナ

コントロールルーム Online Event display ECL hit TOP hit

30 宇宙線テストのオペレーション実際のbeam runと同様に non-expertのexperiment shifterがデータ収集を担当し夜間もデータ取得 Chat tool (rocket chat) でexpert-shifterのcommunication High Level TriggerにてオンラインでCDCのtracking 各検出器のdata qualityのonline monitor Run-control GUI Belle II コントロールルーム Online Event display ECL hit TOP hit CDC track Data quality monitor CDC の ADC スペクトル Online reconstruction した track 函館アリーナ

31 1. イントロダクション 2. Belle II データ読み出しシステム 3. DAQシステムのパフォーマンス 4. 外層検出器の宇宙線試験 5. Belle II 読み出し系のアップグレード計画函館アリーナ

32 Issues to be considered for the Belle II DAQ system Difficulty in maintenance during the entire Belle-II experiment period The number of discontinued parts is increasing. e.g. chipset on a PrPMC card, FIFO and LAN controller on III For older II, it is basically difficult to replace parts according to manufacturer. Four different types of boards(, TTRX, PrPMC, ) should be taken care of. Limitation in the improvement of performance of DAQ A. Bottlenecks of the current readout system CPU usage About 60% -CPU is used at 30kHz L1 trigger rate with 1kB event size/ (=Belle II DAQ target value ) Data transfer speed 1GbE/ B. Bottleneck due to network output of ROPC We need to upgrade the readout system when * luminosity of SuperKEKB exceeds expectations. * Lower threshold of L1 trigger is used or trigger-less DAQ is realized. Depending on throughput, network and HLT farms also need to be upgraded. 函館アリーナ

33 アップグレードの際の境界条件 Readout system FEE b2link Readout GbE-10GbE PC Event builder1 and High level trigger Basic framework of belle2link (Rocket-IO based serial link) should be the same. Otherwise FEE s FW/HW update might be needed. Upgrade like GbE -> 10GbE will be possible, if we upgrade switches. 函館アリーナ

34 種々のオプション FPGA CPU FEE b2link RO RO RO board RO board Ethernet PC HLT (a) -like FEE b2link RO board PCIe PC HLT (b) PCIe FEE b2link RO RO RO RO board board RO board fiber / ATCA backplane RO board HLT (c) 2 step ( Igor-san@15Nov.B2GM ) FEE b2link RO board HLT (d) 1 step 函館アリーナ

35 New readout system = High-density FGPA-based system using utca Data processing speed Fast FPGA-based data processing Data transfer speed 10GbE ( directory connected to a HLT unit ) or 1GbE ( keep readout PCs ) Compact and high-density system high density connector and higher throughput Easier maintenance Currently : 5 s, 5 TTRXs, 5PrPMCs, 20s -> one AMC board (in the case of 20ch/AMC) Schematic view of a new readout board From FEE Patch panel SC SC SC SC SC SC Readout PC/HLT MCH CPU card Advanced Mezzanine Card (AMC) Slow control uatca backplane FTSW 函館アリーナ

36 まとめ Belle II 実験 2018 年 2 月からの phase II run ( 崩壊点検出器以外インストール first collision, beam b.g. 測定 ) に向けて準備が進んでいる Belle II 実験読み出しシステム 7 つのサブ検出器のうち PXD は特殊な読み出し系その他は共通の読み出しシステムを使用読み出しボード () に新たに開発した高速データ受信ボード AtomCPU ボードを搭載して FEE との通信とデータ処理を行う読み出しシステムのパフォーマンス試験 FEE <-> <-> readout PC -> Belle II 実験のトリガーレート (30kHz) で動作することを確認読み出しボードの upgrade を検討中高密度高スループット化函館アリーナ

Express5800/320Fa-L/320Fa-LR

Express5800/320Fa-L/320Fa-LR 7 7 Phoenix BIOS 4.0 Release 6.0.XXXX : CPU=Pentium III Processor XXX MHz 0640K System RAM Passed 0127M Extended RAM Passed WARNING 0212: Keybord Controller Failed. : Press to resume, to setup