Updated Intel Trace Analyzer and Collector (markdown)
[dealii.wiki.git] / Intel-Trace-Analyzer-and-Collector.md
1 # Using Intel Trace Analyzer and Collector (work-in-progress)
2
3 ![itac_example](https://user-images.githubusercontent.com/8023934/41157662-df8546e2-6b26-11e8-8fd2-1ad69ea56605.png)
4
5 ##  links
6
7 - https://software.intel.com/en-us/intel-trace-analyzer/documentation 
8 - https://software.intel.com/en-us/itc-user-and-reference-guide-user-guide 
9 - http://www.prace-ri.eu/IMG/pdf/WP237.pdf 
10 - https://software.intel.com/en-us/intel-trace-analyzer-support/training
11
12 similar tools
13 - http://ipm-hpc.sourceforge.net
14 - http://hpctoolkit.org
15
16 ## Usage Procedure
17
18 Below we consider the case when a user/developer would like to manually measure non-MPI timing of certain regions of the deal.II library and/or user code. This necessitates extra include and link flags.
19
20 ### 0) Setup environment variables
21 If you are using a cluster that uses modules, you should set appropriate environment variables by loading the respective modules. This might look like:
22 ```
23 module load intel64/18.0up02 itac/2018up02 cmake git
24 ```
25 This might already initialize the variables for `Intel Trace Analyzer and Collector`correctly.
26 Otherwise, set the repective environment variables explicitly:
27 ```
28 source <INTEL_INSTALL_DIR>/parallel_studio_xe_<VERSION>.x.xxx/bin/psxevars.sh
29 ```
30
31 ### 1.a) Build deal.II
32
33 ```
34 module load intel64/18.0up02 itac/2018up02 cmake git
35 cmake ../ -DCMAKE_CXX_FLAGS:STRING="-g -trace -O2 -march=native" -DDEAL_II_LINKER_FLAGS="-trace" -DDEAL_II_INCLUDE_DIRS="/apps/intel/ComposerXE2018/itac/2018.2.020/include" -DDEAL_II_DEFINITIONS="USE_VT"
36 ```
37 Notes:
38 * Avoid specifying MPI libraries manually, this ruins link sequence set up by compiler flag `–trace`.
39
40 Now you can add timers to required parts of the library
41 ```
42 #ifdef USE_VT
43 #include <VT.h>
44 #endif
45 ...
46 #ifdef USE_VT
47   VT_Region region("my_name", “my_group", __FILE__, __LINE__);
48 #endif
49 <some-function>
50 #ifdef USE_VT
51   region.end();
52 #endif
53 ```
54
55 Finally build the library
56 ```
57 make all -j20
58 ```
59
60 ### 1.b) Build downstream library/project
61
62 Add to `CMakeList.txt`
63 ```
64 IF (DEFINED USE_VT)
65   ADD_DEFINITIONS(-DUSE_VT)
66 ENDIF (DEFINED USE_VT)
67 ```
68 and then build the user project/library as usual
69 ```
70 cmake ../ -DDEAL_II_DIR=/path/to/dealii -DUSE_VT
71 make all -j20
72 ```
73 Similar to the above, you can manually add timers to certain parts of the project.
74
75 ### 2. Prepare a configuration file  `trace.conf`
76 ```
77 # Log file
78 LOGFILE-NAME my_trace.stf
79 LOGFILE-FORMAT STF
80 # disable all MPI activity
81 ACTIVITY MPI OFF
82 # enable all bcasts, recvs and sends
83 SYMBOL MPI_WAITALL ON
84 SYMBOL MPI_IRECV ON
85 SYMBOL MPI_ISEND ON
86 SYMBOL MPI_BARRIER ON
87 SYMBOL MPI_ALLREDUCE ON
88 # enable all activities in the Application class 
89 ACTIVITY Application ON
90
91 STATE Application:* UNFOLD
92 STATE lib*:* FOLD
93 STATE libdeal*:* UNFOLD
94 ```
95
96 ### 3. Run the application (usually by submitting an interactive job to the queue)
97
98 ```
99 qsub -l nodes=1:ppn=40,walltime=01:00:00 -I
100 module load intel64/18.0up02 itac/2018up02
101 export VT_CONFIG=/path/to/trace.conf
102 mpirun -trace -np 20 my_executable
103 ```
104
105 You should see at the end: 
106 ```
107 [0] Intel(R) Trace Collector INFO: Writing tracefile….
108 ```
109
110 ### 4. GUI to analyze:
111 ```
112 $ module load itac/2018up02
113 $ traceanalyzer my_trace.stf
114 ```

In the beginning the Universe was created. This has made a lot of people very angry and has been widely regarded as a bad move.

Douglas Adams


Typeset in Trocchi and Trocchi Bold Sans Serif.