ca87422242b32b0c89287e5d25a205cff697de0e
[dealii.wiki.git] / Intel-Trace-Analyzer-and-Collector.md
1 # Using Intel Trace Analyzer and Collector
2
3 ![itac_example](https://user-images.githubusercontent.com/8023934/41157662-df8546e2-6b26-11e8-8fd2-1ad69ea56605.png)
4
5 ##  links
6
7 - https://software.intel.com/en-us/intel-trace-analyzer/documentation 
8 - https://software.intel.com/en-us/itc-user-and-reference-guide-user-guide 
9 - http://www.prace-ri.eu/IMG/pdf/WP237.pdf 
10 - https://software.intel.com/en-us/intel-trace-analyzer-support/training
11
12 similar tools
13 - http://ipm-hpc.sourceforge.net
14 - http://hpctoolkit.org
15
16 ## Usage Procedure
17
18 Below we consider the case when a user/developer would like to manually measure non-MPI timing of certain regions of the deal.II library and/or user code. This necessitates extra include and link flags.
19
20 ### 0) Setup environment variables
21 If you are using a cluster that uses modules, you should set appropriate environment variables by loading the respective modules. This might look like:
22 ```
23 module load intel64/18.0up02 itac/2018up02 cmake git
24 ```
25 This might already initialize the variables for `Intel Trace Analyzer and Collector` (`ITAC`) correctly.
26 Otherwise, set the repective environment variables explicitly:
27 ```
28 source <INTEL_INSTALL_DIR>/parallel_studio_xe_<VERSION>.x.xxx/bin/psxevars.sh
29 ```
30
31 ### 1.a) Build deal.II
32
33 ```
34 cmake ../ -DCMAKE_CXX_FLAGS:STRING="-g -trace -O2 -march=native" -DDEAL_II_LINKER_FLAGS="-trace" -DDEAL_II_INCLUDE_DIRS="$ITC_INC" -DDEAL_II_USER_INCLUDE_DIRS="$ITC_INC" -DDEAL_II_DEFINITIONS="USE_VT" -DDEAL_II_USER_DEFINITIONS="USE_VT"
35 ```
36 where `ITC_INC` is environment variable with `ITAC` include directory, i.e. `/apps/intel/ComposerXE2018/itac/2018.2.020/include`.
37
38 Notes:
39 * Avoid specifying MPI libraries manually, this ruins link sequence set up by compiler flag `–trace`.
40
41 Now you can add timers to required parts of the library
42 ```
43 #ifdef USE_VT
44 #include <VT.h>
45 #endif
46 ...
47 #ifdef USE_VT
48   VT_Region region("my_name", “my_group", __FILE__, __LINE__);
49 #endif
50 <some-function>
51 #ifdef USE_VT
52   region.end();
53 #endif
54 ```
55
56 Finally build the library
57 ```
58 make all -j20
59 ```
60
61 ### 1.b) Build downstream library/project
62
63 Configure the user project/library as usual
64 ```
65 cmake ../ -DDEAL_II_DIR=/path/to/dealii
66 ```
67
68 Similar to the above, you can manually add timers to certain parts of the project and then build it
69
70 ```
71 make all -j20
72 ```
73
74 ### 2. Prepare a configuration file  `trace.conf`
75 ```
76 # Log file
77 LOGFILE-NAME my_trace.stf
78 LOGFILE-FORMAT STF
79 # disable all MPI activity
80 ACTIVITY MPI OFF
81 # enable all bcasts, recvs and sends
82 SYMBOL MPI_WAITALL ON
83 SYMBOL MPI_IRECV ON
84 SYMBOL MPI_ISEND ON
85 SYMBOL MPI_BARRIER ON
86 SYMBOL MPI_ALLREDUCE ON
87 # enable all activities in the Application class 
88 ACTIVITY Application ON
89
90 STATE Application:* UNFOLD
91 STATE lib*:* FOLD
92 STATE libdeal*:* UNFOLD
93 ```
94
95 ### 3. Run the application (usually by submitting an interactive job to the queue)
96
97 ```
98 qsub -l nodes=1:ppn=40,walltime=01:00:00 -I
99 module load intel64/18.0up02 itac/2018up02
100 export VT_CONFIG=/path/to/trace.conf
101 mpirun -trace -np 20 my_executable
102 ```
103
104 You should see at the end: 
105 ```
106 [0] Intel(R) Trace Collector INFO: Writing tracefile….
107 ```
108
109 ### 4. GUI to analyze:
110 ```
111 $ module load itac/2018up02
112 $ traceanalyzer my_trace.stf
113 ```

In the beginning the Universe was created. This has made a lot of people very angry and has been widely regarded as a bad move.

Douglas Adams


Typeset in Trocchi and Trocchi Bold Sans Serif.