Scripts

Par Alexandre Ganachaud — Dernière modification 03/03/2014 10:54 — Historique

Scripts for oxygen calibration (originally PMEL/C. Johnson)

CTD oxygen calibration procedure

A. Ganachaud

Data format

Basic: Netcdf OceanSITES (US-IMAGO)

Attributs lower case and dates format IS08660,
l'attribut global 'data_type' donne le type de données: profile, trajectory ou time-series,
ex: data-type= "OceanSITES profile data"
Dimensions TIME et DEPTH au lieu de PROFILE et PRES
Variables 1D: TIME (jour julien décimal depuis 1950) a la place de DAYD, et LATITUDE et LONGITUDE LATX et LONX)

Units

Reliable units from chemistry excel sheet are in ml/l
They are converted into umol/kg during merge-ctd-btl because one need to use the ctd salinity information.
Robbins' Routine updated Dec 29 2014 as follows: 1 μmol O2 = .022391 ml (from CCHDO and ODV) (decrease of 0.05%)

Flags OceanSites

From IGOSS/ OceanSITES (includes CTD bottle/samples)

0=no_qc_performed
1=good_data
2=probably_good_data
3=bad_data_that_are_potentially_correctable
4=bad_data
5=value_changed
6=not_used
7=nominal_value
8=interpolated_value
9=missing_value

Flags WHP-CTD

From CCHDO

1 Not calibrated.
2 Acceptable measurement.
3 Questionable measurement.
4 Bad measurement.
5 Not reported.
6 Interpolated over >2 dbar interval.
7 Despiked.
8 Not assigned for CTD data.
9 Not sampled.

Flags WHP-Bottle themselves

From CCHDO

1 Bottle information unavailable.
2 No problems noted.
3 Leaking.
4 Did not trip correctly.
5 Not reported.
6 Significant discrepancy in measured values between Gerard and Niskin bottles.
7 Unknown problem.
8 Pair did not trip correctly. Note that the Niskin bottle can trip at an unplanned depth while the Gerard trips correctly and vice versa.
9 Samples not drawn from this bottle.

Flags WHP-Bottle samples

From WHP/CCHDO

1 Sample for this measurement was drawn from water bottle but analysis not received. Note that if water is drawn for any measurement from a water bottle, the quality flag for that parameter must be set equal to 1 initially to ensure that all water samples are accounted for.
2 Acceptable measurement.
3 Questionable measurement.
4 Bad measurement.
5 Not reported.
6 Mean of replicate measurements (Number of replicates should be specified in the -.DOC file and replicate data tabulated).
7 Manual chromatographic peak measurement.
8 Irregular digital chromatographic peak integration.
9 Sample not drawn for this measurement from this bottle.

Flags ODV

here (with correspondence among the different conventions)

Procedure

Reference: Swift, J. H., 2010. IOCCO Report 14, ICPO Publication series No 134 (Check Ushida et al.)

Recommendations by K.Mctaggard:
"Group with as big a group as possible so that the residuals are small
(2 umol/kg). Coefficients Slope and bias can vary, Oxcor and Tcor and
lag should remain stable (lag=2 to 7 seconds generally)"

During the cruise

if possible:

1- visually check the descending profile versus bottle chemical data (cmp_btl_profile.m)

2- run merge_ctdbtl_chimie.m (parameters to be defined in o2_params.m) station by station, then for all stations

After the cruise

SBE-43 Procedure (A. Ganachaud, June 2008)

ctd-procedure

o2_params.m

Contains all parameters to run the different scripts. For all but run_oxygen_cal_ml, the script will call o2_params first to load the parameters. For calibration step, need to run o2_params with parameter p_calib=1.

Step 1: merge_ctdbtl_chimie.m

%FOR GIVEN STATION GROUP:

% READ ROSETTE FILE (.BTL or _btl.nc)

% READ OXYGEN EXCEL SHEET

% FIND BOTTLE CLOSE TO ROSETTE1 BOTTLE and ASSOCIATE O2 (WARNING IF 2 CLOSE BOTTLES OR NO MATCH)

%WRITE .CAL and _cal.mat bottle files with all stations for match_sgn

Input:

This uses read_xls_o2.m to read xls file. For PANDORA, each xls contained group of 10 stations
Uses the .btl files with ctd rosette information (e.g., pandora_btl.nc)

Output: "cal" file with all samples/rosette info: both ascii (.cal) and mat (_cal). Ascii has each column as follows:

btl.stat=p(:,1); %station
btl.cast=p(:,2); %cast
btl.btlid=p(:,3);%bottle id
btl.upres=p(:,4); %u = up ctd pressure
btl.utemp68=p(:,5);% up ctd temperature
btl.bsali=p(:,6); %salinity from samples
btl.usali=p(:,7); % up ctd salinity
btl.flask=p(:,8); %flacon id
btl.uoxv=p(:,9); %up ctd o2 voltage
btl.uoxum=p(:,10); %up ctd o2 umol/kg
btl.wnklo2um=p(:,11); %oxygen from samples (umol/kg)

Step 2: match_sgn.m

%READ THE .CAL BOTTLE FILE

%READ DOWNCAST PROFILES

%FIND BOTTLE DENSITIES ON DOWNCAST PROFILE

%ASSIGN O2 SAMPLE DATA TO CTD ON SAME DENSITY FROM DOWNCAST

%GRAPHICS WITH EACH PROFILE THEN ENSEMBLE RESIDUALS

Notes:

Output: Califiles/oxydata_<cname>p.mat: contain btl.d<variables>: d=downcast that are density adjusted. Also contains btl.d<variables>P that are at constant pressure.
Run station by station first to watch for irregularities: unstable for homogeneous densities; gaps or glitches in CTD profile; spurious upcast/.blt data
Hard coded maximum 50m for sigma-match. If more, goes back to pressure match
Watch effect of filtering dvdt: is 2 to 8-m normal for density inversion corrections ? Yes: GJ got generally 5-10; up to 20m
Graphics (p_final=1): matching effect on residuals: match_sig_oxyg_1.png; match_sig_pres.png. Use p_plot_upcastbottles=1 to also see the upcast oxygen deviations.
Variables:

ctd.<field>: for each profile, contains some or all of these (example value/dimensions):

stat: 10
cast: 7
dayd: 184.8635
latx: -11.9993
lonx: 162.9988
dpres: [98x1 double] -down pressure
dtemp68: [98x1 double] -down temperature (on scale ITS68 for use with sw_ routines)
dsali: [98x1 double] -down salinity
dsali_qc: [98x1 double] -down salinity quality flag (not used)
doxv: [98x1 double] -down oxygen voltage
ddvdt: [98x1 double] -down oxygen voltage derivative
doxum: [98x1 double] -down raw ctd oxygen in umol/kg
dosat: [98x1 double] -down oxygen saturation
dsdvdt: [98x1 double] -down filtered oxygen derivative
oxcal: [98x1 double] -down raw calibrated oxygen
oxcal_qc: [98x1 double] -down raw calibrated oxygen quality flag (not used)
oxcali: [98x1 double] -down raw calibrated oxygen unspiked
oxcali_qc: [98x1 double] -down raw calibrated oxygen unspiked quality flag
dtemp90: [98x1 double] -down temperature (on scale ITS90)
dsalii: [98x1 double] -down salinity unspiked
dsalii_qc: [98x1 double] -down salinity unspiked quality flag

Step 3: run_oxygen_cal_ml.m

Inputs:

oxygdata_<cruise>p.mat / contains btl.xxxx

This is run from o2_params using p_calib=1
Recommended to use PMEL equation (see correspondance with C. Johnson, 2013):
Soc*(V+Voff+T20_DO*exp(DI*P+D2*T).*dVdt).*os.*exp(Tcor*T).*exp(Pcor*P./(273.15+T))
Need to ask SEABIRD for appropriate calibration sheet with Tcor
Trials: try switches and groupings to get better residuals. Start with iswitch=0/wswitch=0. Use iswitch if linear trend in residuals with station; use wswitch if bias at depths.
Outputs

Save in "final" the calibration matrix with coefficients for each profile
Graphics: o2cal_residuals_stat_1_82_swslp_0_0.png: residuals by depths or station (csuf/wswitch/iswitch); o2cal_residuals_stat_1_82_swslp_0_0-bydepthrange.png: same, but for specified depth ranges.
Bottle output files:

sample_with_qc_%s_is%d_ws%d_%s_upres.txt: bottle data on bottle sample pressure with quality flags according to the ones rejected (flag 3) by the calibration process
sample_with_qc_%s_is%d_ws%d_%s_dpres.txt: same but pressure are matched pressures (use for graphics for calctd/cleanox next)

Step 4: calctd.m / cleanox.m

Get and calibrate ctd profiles (whole profiles) based on calmat ('final')
Create subdirectory 'Caliprofiles' first for output for each profile (.mat)
Note that profiles are filtered in the code, (p_filter, basically 9m)
Procedure:

Step 1: run with p_cleanox / p_cleansalt=0 /p_cleantemp=0: and p_onlyonecast=1.

This will calibrate all main ctd profiles (no secondary casts with no bottles)
Step to check the bad bottles in o2 and salinity; record those in our excel bottle summary sheet
Do the same for all station groups

Step 2: run_plotTS.m (see below)

This will show theta-S comparison between CTD calibrated and bottles.
Do that for all station groups together
If see shifts with station groups, move back to calibration

Step 3: run with p_onlyonecast=0 p_cleanox=1 to unspike all profiles
Step 4: run again with p_cleansalt=1 to unspike salinities (and overwrite output files)
Step 5: run again with p_cleantemp=1 to unspike temperatures (and overwrite output files)

p_cleanox=0:

check main casts (with oxygen samples); take notes
spot outlier bottles; detect leaks
graphic output with calibrated profiles and samples
go to step 5 to verify deep theta-O relationships
if satisfactory, unspike and verify using cleanox option

the p_cleanox=1: call function cleanvar.m for oxygen

Flags everything at 2 ("acceptable") (see whp convention)
Interactively visualize and interpolate (flag 6) / eliminate data (flag 9): watch both menu/window title and prompt; need to type "return" to continue program for interpolation.

p_cleansalt / p_cleantemp=1: call function cleanvar.m for salt and temp

Flags everything at 2 ("acceptable") (see whp convention)
Interactively visualize and interpolate (flag 6) / eliminate data (flag 9): watch both menu/window title and prompt; need to type "return" to continue program for interpolation.

Graphics:

Current graphics include bottle data overlay, even from other cast; in green are the previous casts; check also the T/S for anomalies
Generates and print similar graphic for each cast with bottles overlain (red cross for the current cast; blue cross if bottles are from another cast). Graphics in dirjpg/direps, ex: ox_cal_pandora_STN035_CAST04.png

Outputs:

as ctd or actd fields:

oxical: calibrated oxygen
oxicali: unspiked oxygen
dsalii: unspiked salinity
and same with extension _qc for quality flag

individual profiles (ctd.xx ) in Caliprofiles/ox_cal_pandora_STN010_CAST03.mat
all profiles in (actd.xx ) Califiles/all-o2-calibrated_pandora_STN001_STN010.mat

Graphics: as we go but also use plot_calctd_all.m for a global loop to generate graphics after completed calctd/cleanox

Step 5: run_plotTS.m

interactive/reprogram as needed: this calls several times plotTS routines
For selected deep layers, plots theta-S and theta/O2 for groups of stations; overlay bottles
Scope is to detect any steps in calibrated oxygen that are not in bottle data
Graphics: ox_cal_theta_S-O.png (watch out the name of the graphic as it is based on the last parameters used)

Optional graphics

plot_section_ctd.m

Subsampling

filter_subsample_all.m : apply median filter / Hanning filter at p_filter and subsample every 5m for common usage and better graphics
Input: individual ctd profiles
Output:

global arrays (actd.) with one single profile/station; variables take suffix f (filtered)
individual cast profiles (ctd.; subdirectory FilteredOP/); variables take suffix f (filtered)

END OF STANDARD ROUTINES

Previous, cruise-specific routines h:

calctd_sec04

merge with calibrated T,S
interpolate (unspike) oxygen if necessary

cleanox_sec04.m:
gets calibrated profiles; show/interpolate little gaps/interpolate manually/compare with other cruises/upcasts
save interpolated data / flagged i

merge_ctdcal_oxycal.m:
merges ctd T,S data that were calibrated independently with calibrated oxygen data; show oxygen bottles overlay.

SBE-43 procedure with G. Johnson's initial routines

match_sgn_315p_new.m
load p18_315o_pri.cal;
textread('dlp_stnc_315.txt')
textread each file in there (one per station)
save oxydata_315p z
write 'p18_315o_pri.clo'
figure sdvdt

Prepares the .clo file by take the downcast ctd and matching the upcast oxygen bottle of the same density. One value per bottle in the output (.clo) file.

function calmatrix=run_oxygen_cal_ml(stas,iswitch,wswitch);
load oxydata_315p;
load temp_oxydata_ml
save temp_oxydata_ml z outliers c nc wswitch stas cc

Finds the appropriate parameters to compute oxygen from the equation. It will remain to effectively compute it!

function r=slopeandbias_ml(sb);
load temp_oxystaadjust_ml

function r=oxygen_cal_staslope_ml(c);
load temp_oxydata_ml

function r=oxygen_cal_ml(c);
load temp_oxydata_ml

Previous instructions for SBE-13

Run merge_ctdbtl_chimie.m to:

%FOR GIVEN STATION GROUP:
% READ ROSETTE FILE (.BTL)
% READ OXYGEN EXCEL SHEET
% GROUP THEM IN A CLO FILE
%0-READ ROSETTE DATA
%1-READ BOTTLE DATA
%2-FIND BOTTLE CLOSE TO ROSETTE BOTTLE and ASSOCIATE O2 (WARNING IF 2 CLOSE BOTTLESOR NO MATCH
%3-LOAD DOWN PROFILE
%4-CALCULATE DENSITY
%5-FIND PRESSURE P1 FOR WHICH DENSITY = BOTTLE DENSITY
%6-SET OTHER QUANTITIES FROM DOWNCAST AND FILTER DOCDT (5 DBAR MEDIAN)
%7-WRITE P1 OX OC DOCDT FROM DOWNCAST IN FINAL BOTTLE FILE (.dbtl)

run make_clo_file.m to build the clo file for groups of stations for calibration

gedit oxfitmr.input; run oxfitmr (actually ./oxfitmr<oxfitmr.input) OR run all from calox.m which creates input file to oxfitmr and call oxfitmr (under matlab-linux)

Note the regression coefficients for all stations from .log and regression result from histo

Plot cumulative residuals to find changes suggesting station grouping

Once groups are made, final oxygen calibration using oxycalc.m on downcast profiles

put in coefficients as function of station number from web page

run cleanox to check for spikes

run oxdwnp2_.m for comparing again bottle and ctd data and making final statistics

run merge_ctdcal_oxycal.m to re-write calibrated T/S data from data_final adding calibrated oxygen and flags

run plot_ctd (subdir Plotcampagne)

Actions sur le document

Laboratoire d’Etudes en Géophysique et Océanographie Spatiales