ffdddbc0bdf11efc4c8576d3f2402ab631224521
[dealii.wiki.git] / Intel-Trace-Analyzer-and-Collector.md
1 # Using Intel Trace Analyzer and Collector (work-in-progress)
2
3 ![itac_example](https://user-images.githubusercontent.com/8023934/41157662-df8546e2-6b26-11e8-8fd2-1ad69ea56605.png)
4
5 ##  links
6
7 - https://software.intel.com/en-us/intel-trace-analyzer/documentation 
8 - https://software.intel.com/en-us/itc-user-and-reference-guide-user-guide 
9 - http://www.prace-ri.eu/IMG/pdf/WP237.pdf 
10 - https://software.intel.com/en-us/intel-trace-analyzer-support/training
11
12 similar tools
13 - http://ipm-hpc.sourceforge.net
14 - http://hpctoolkit.org
15
16 ## Usage Procedure
17
18 Below we consider the case when a user/developer would like to manually measure non-MPI timing of certain regions of the deal.II library and/or user code. This necessitates extra include and link flags.
19
20 ### 1.a) Build deal.II
21
22 ```
23 module load intel64/18.0up02 itac/2018up02 cmake git
24 cmake ../ -DCMAKE_CXX_FLAGS:STRING="-g -trace -O2 -march=native" -DDEAL_II_LINKER_FLAGS="-trace" -DDEAL_II_INCLUDE_DIRS="/apps/intel/ComposerXE2018/itac/2018.2.020/include" -DDEAL_II_DEFINITIONS="USE_VT"
25 ```
26 Notes:
27 * Avoid specifying MPI libraries manually, this ruins link sequence set up by compiler flag `–trace`.
28 * adjust `intel` modules to fit your setup.
29
30 Now you can add timers to required parts of the library
31 ```
32 #ifdef USE_VT
33 #include <VT.h>
34 #endif
35 ...
36 #ifdef USE_VT
37   VT_Region region("my_name", “my_group", __FILE__, __LINE__);
38 #endif
39 <some-function>
40 #ifdef USE_VT
41   region.end();
42 #endif
43 ```
44
45 Finally build the library
46 ```
47 make all -j20
48 ```
49
50 ### 1.b) Build downstream library/project
51
52 Add to `CMakeList.txt`
53 ```
54 IF (DEFINED USE_VT)
55   ADD_DEFINITIONS(-DUSE_VT)
56 ENDIF (DEFINED USE_VT)
57 ```
58 and then build the user project/library as usual
59 ```
60 cmake ../ -DDEAL_II_DIR=/path/to/dealii -DUSE_VT
61 make all -j20
62 ```
63 Similar to the above, you can manually add timers to certain parts of the project.
64
65 ### 2. Prepare a configuration file  `trace.conf`
66 ```
67 # Log file
68 LOGFILE-NAME my_trace.stf
69 LOGFILE-FORMAT STF
70 # disable all MPI activity
71 ACTIVITY MPI OFF
72 # enable all bcasts, recvs and sends
73 SYMBOL MPI_WAITALL ON
74 SYMBOL MPI_IRECV ON
75 SYMBOL MPI_ISEND ON
76 SYMBOL MPI_BARRIER ON
77 SYMBOL MPI_ALLREDUCE ON
78 # enable all activities in the Application class 
79 ACTIVITY Application ON
80
81 STATE Application:* UNFOLD
82 STATE lib*:* FOLD
83 STATE libdeal*:* UNFOLD
84 ```
85
86 ### 3. Run the application (usually by submitting an interactive job to the queue)
87
88 ```
89 qsub -l nodes=1:ppn=40,walltime=01:00:00 -I
90 module load intel64/18.0up02 itac/2018up02
91 export VT_CONFIG=/path/to/trace.conf
92 mpirun -trace -np 20 my_executable
93 ```
94
95 You should see at the end: 
96 ```
97 [0] Intel(R) Trace Collector INFO: Writing tracefile….
98 ```
99
100 ### 4. GUI to analyze:
101 ```
102 $ module load itac/2018up02
103 $ traceanalyzer my_trace.stf
104 ```

In the beginning the Universe was created. This has made a lot of people very angry and has been widely regarded as a bad move.

Douglas Adams


Typeset in Trocchi and Trocchi Bold Sans Serif.