{9} Assigned, Active Tickets by Owner (Full Description) (114 matches)

This is just to check how the email notification works

The current CIM is enormously complex, and in the process of developing the CMIP5 instances (and in the development of BFG support etc), we have (and will) learn a goodly amount.

We also need to go to ISO19156 compliance as well as better ISO compliance in CIM2.0

Attempts to simplify, along with issues to be dealt with, and plans, should be documented on the accompanying  web page

The home page should have more complete usage instructions, along with diagrams etc.

We need to rejig the feed code so that we can send document sets (formerly known as record sets) to Sylvia. We also need to ensure that the DocumentSet feed is updated if any of the pieces are updated.

Something to worry about in this context is that the model feed is the "unconfigured" model feed. The existing simulation code is where we get the configured model! See this ticket's web page and the ticket:888 web page.

We need a new term to describe the partial fulfilment of a spatiotemporal constraint.
Users need to be able to select this term from the list of conformance CV that drops down under
"How this simulation Conformed:"

suggest: Partly Conforms

I have been testing what happens if you try deleting, references, files and parties.

When I attempt to delete a reference I get an error 500 message or ('NoneType' object is not iterable) even if the reference is not linked to other parts of the questionnaire

The questionnaire won't let me delete a file or party if it is used in other places. :-)

Create graphics that show users how to navigate the questionnaire

• Somewhere before starting the questionnaire, have a list of the things users’ll need to collect to fill out the questionnaire. Should this be on the CMIP5 site?

• Identify one person in each modelling group who is responsible for filling out the questionnaire and organise a mail list with all these people and support people. Keep updating the faqs in the questionnaire help pages.

• Support group needs to decide frequency of dfn sessions for questionnaire training.

• Coordination of support with official CMIP5 data management process? This is a conversation in context of GO-ESSP-TECH.

Re-think how we document partially-overlapping and non-overlapping "ensembles" for CMIP5 such as the 1.1 decadal hindcasts and possiblly also the TAMIP experiments #683

It is not necessary to implement a solution to this for V1.0, but we will need an agreed strategy as it may impact CMOR and the DRS.

See Discussion at tickets/684

Proposed naming convention for TAMIP: TAMIP
also includes application to the decadal hindcast/forecast experiments

How will the ensemble page in the questionnaire look? - #712

We need to add "IBM2003" compiler to the platform page what other additions need to be made?

Other platform tickets #905

• Issue of copying questionnaire instances between institutions. We’ve already had requests to be able to do this. We could use an examples area where people can find demo descriptions, or people could ask us to copy from one institution to another.

• How do we handle experiment layout to make it easy for people to submit extended experiments? Generate a new DoI for an extension?

Laurent Fairhead wrote: 
Hello all, 

Sébastien, Marie-Alice and I went to Sandro's presentation. It was mainly about how they deal with querying existing databases in a distributed environment using the EGEE grid. I guess their work overlaps  some of what we're wishing to do in WP4 (except the harvested CIM database that we're planning). Sébastien had a talk with Sandro Fiore after the presentation and he's apparently ready to interact with us and would be willing to develop functionalities in his system for our needs. 

Their web site is here: 

http://grelc.unile.it/home.php 

Cheers 
Laurent 

Valcke Sophie wrote: 

Hi, 

CMCC  (and Sandro Fiore) is highly involved in the IS-ENES proposal and know perfectly well about Metafor. We had a lot of discussion with them during the writing of the proposal and I think Michael  and Martin Juckes (from BADC?) interacted with CMCC, technically speaking. Of course, I think it would be a good idea for Metafor WP4, WP5 and WP6 to liaise with them. 

  regards, 
   Sophie 

Eric Guilyardi wrote: 


----------------------------------------- 
CMCC Grid Metadata Handling System 

The next generation of climate modeling researchers will face a critical challenge, i.e., dealing with increasingly complex simulation models and huge quantities of related datasets, which are already too massive for current storage, manipulation, archiving, navigation, and retrieval capabilities. From the data grid perspective a key element to discover, manage and access huge and distributed amount of data will be the metadata handling framework. While from the grid data handling perspective several solutions are already available and can be adopted, for metadata management centralized solutions are usually proposed. What we propose for the management of a distributed ES production activity is the CMCC Metadata Handling System (a data grid solution leveraging distributed and P2P data grid services focusing on metadata management and access). From our point of view, centralized solutions are not feasible and are not able to address availability, scalability, robustness and efficiency at such large scale. Despite the classical approaches, data-grid-enabled solutions are able to greatly address scalability (users, data, etc.), transparency (access, integration, presentation) and efficiency (performance). The Grid Metadata Handling System we propose is able to provide both access to and integration of metadata stored in different and widespread data sources providing a strong data virtualization layer in grid. Moreover we must take into account that (i) such a technological solution for distributed metadata management leverages on standards, emerging specification, etc. (ii) supports role-based management (based on VOMS), which increases flexibility and scalability (iii) provides full support for Grid Security Infrastructure, which means (authorization, mutual authentication, data integrity, data confidentiality, etc.); (iv) is based on P2P grid protocols/services, fully addressing interoperability, data integration and transparency. 

----------------------------------------- 
GRelC Project 

Grids encourage and promote the publication, sharing and integration of scientific data, distributed across Virtual Organizations. Scientists and researchers (bioinformatics, astrophysics, etc.) work on huge, complex and growing datasets. The complexity of data management within a grid environment comes from the distribution, heterogeneity and number of data sources. Along with coarse grained services (basically grid storages, replica services, storage resource managers, etc), there is a strong interest on fine grained ones concerning, for instance, grid-database access and management. Moreover, as grid computing, technologies and standards evolve, more mature environment (production grids such as EGEE) become available for production based activities and tools/services able to access in grid to relational databases are also strongly required. 
Within the proposed talk we will describe in detail the Grid Relational Catalog (GRelC) Project, an integrated environment for grid database management, highlighting the vision/approach, architecture, components, services and technological issues. 
The key topic will be the GRelC Data Access and Integration Service. 
The GRelC DAIS is a GSI/VOMS enabled web service addressing extreme performance, interoperability and security. It efficiently, securely and transparently manage databases on the grid across VOs, with regard to emerging and consolidated grid standards and specifications as well as production grid middleware (gLite & Globus). 
It provides a uniform access interface, in grid, both to access and integrate relational (Mysql, Oracle, Postgresql, IBM/DB2, SQLite) and non-relational data sources (XML DB engines such as eXist, XIndice and libxml2 based documents). 

consider in page guidance videos as well

CERFACS

Simulation Tab: Required Inputs and Couplings Frame (title)

Add explanatory text under "Required Inputs and Coupling" along the lines of:

"List of Inputs Needed by the Components of the Model that ran for this Simulation that have to be Resolved here"

CERFACS

Simulation Tab: Conformance Frame (text)

<(You can modify these using the button on the left hand menu.)>. Which button? replace by: <...using the "Conformance" button...>

• Make sure that modellers are aware of contact policy for email addresses put into the questionnaire. Suggest that they might like to use a generic institutional email address. o ESG will display whatever contact information Metafor will provide, and can attach roles to contact information. We need to consult with the modelling groups about the contact information they want to display.

There are about 1500 unused primary key indices in the postgres database. Why?

CERFACS

All Tabs

CERFACS Would like to see a message apear that warns users that data will be lost if they navigate away from a page without saving. 

This could become tiresome once users are used to editing the questionnaire. I think this would work best for users if the warning mode can be turned off.

GML serialisation does not work yet (due to non-conformance of UML). Plan to use this ticket to document any changes I make to CIM 1.91

We require a formal approval process for the questionnaire.

When the questionnaire is ready, we will give it to Karl Taylor, Ron Stouffer et al. for approval before it is released

• Eric and Bryan to contact Karl and Ron to define an approval process.

I'm working with the questionnaire in Gerry's branch. I just described a model mod as a parameter change but it appears as type=codemod in the list of model mods - see screen shots.
 description  list

Give users read only access to all modelling centres in the questionnaire.  Users only have write access to the test area and to their own modelling centre.

Hi again MP :-)

Could you let me know what is mandatory in the grid part of the questionnaire and what is optional. I need this to be able to tell if someone has completed the page or not.

For example, must people provide a value for "latMin"?

Thanks

-- Rupert

The questionnaire needs to export enough information to support a mapping between the "meta4 metadata" and the published (realm) level datasets.

This needs to support the metadata creator wanting to match a particular simulation description version to a particular realm dataset.

There are a couple of obvious use cases:

1. I've just written a simulation description, but which one of the two versions of data on the disk does it point to? How do I indicate that?
2. I've just made a change to some data, published a new version. The simulation hasn't changed, but the data associated with it has. Where and how do I fix that?
   • I might want the simulation description to point to both, or just one. The latter case might occur when I want the simulation description to eventually be the DOI target: it can only be associated with one version on disk. However, if it isn't associated with a DOI, there is no reason why it can't be associated with a number of versions of data on disk (associated with fixing data handling mistakes, if the simulation has been rerun, the simulation description ought to need updating!)

In practice the only way this can be done is in the simulation description, which has the mechanism under the hood, but no part of the simulation view deals with it. We need to fix it.

During a demo of the ESG display to a scientist at a conference, it was noted that this person (who is entering info for GISS), wanted to have an easy to up date the metadata submitted well after submission.

They want to be able to tell what items were in fact updated.

Currently, in the ensemble description page, the questionnaire will ask details regarding the changes/mods for ensemble member 1. Does this make sense, i.e what would be entered here that won't have already been entered at the simulation level (apart from the DRS name)? This can be removed for member 1 if there is general consensus that this isn't necessary.

From year 1 meeting:

o Work on associated diagnostic variables

 http://q.cmip5.ceda.ac.uk/cmip5/15/edit/file/181/list/

I described a new file and attempted to associate it with an experiment but when I save the file description the associations I selected disappear.

A useful usability feature would be to allow the user within the questionnaire page to reference a previously described component (or grid component) from another model (within the same centre) instaed of having to reenter information. This is in addition to the 'copy model' button already in existance which may not always solve the issue. One proposed solution is that at each realm level component (or horiz/vertical level in grid), there is a select box to say 'using already described component' with a dropdown box that lists currently available model/components for that particular centre. On choosing this, the empty form fields for new information would subsequently be removed from the page.

The questionnaire xould be improved significantly, both from an ease of use and time point of view, by introducing a pop-up box feature when a user needs to add/attach a new/existing entity to a given resource, e.g attaching a new responsible party to a component. This feature would be ajax-based, and would mean that the user is not required to leave the current page on which he is working, and thus minimising unnecessary navigation across pages.

The questionnaire could be improved through changes to its general layout. In particular, this would involve two primary alterations:

1) reorganise the tabs at the top to separate out 'ancillary' tabs from 'process' tabs, e.g.

right-aligned= files/people/refs etc left-aligned = workflow, i.e models, simulations

2) have a horizontal menu bar near the top at the page level for the validate/view/export buttons, as opposed to lower down on the left hand panel.

The simulation duration start date should be mandatory on the simulation page but is optional. Note, this value may be empty when we load the experiments (as some have an open date range), so an optional value must be supported to start off with, but a value must be provided before the simulation page is saved by a user.

The latest translator code conforms to CIM1.5. The viewer is failing from the simulation page with the following error:

Request Method: GET Request URL: http://localhost:8001/cmip5/45/simulation/doc/3/cimView/ Exception Type: IndexError Exception Value:

list index out of range

Exception Location: /home/rupert/proj/metafor/metaforSVN/cmip5q/cmip5q/cmip5q/protoq/cimHandling.py in cimViewDoc, line 310 Python Executable: /usr/bin/python Python Version: 2.6.5

Traceback  Switch to copy-and-paste view

• /usr/local/lib/python2.6/dist-packages/Django-1.1.1-py2.6.egg/django/core/handlers/base.pyinget_response 85. # Apply view middleware
  85. for middleware_method in self._view_middleware:
  86. response = middleware_method(request, callback, callback_args, callback_kwargs)
  87. if response:
  88. return response
  90. try:
  91. response = callback(request, *callback_args, callback_kwargs) ...
  92. except Exception, e:
  93. # If the view raised an exception, run it through exception
  94. # middleware, and if the exception middleware returns a
  95. # response, use that. Otherwise, reraise the exception.
  96. for middleware_method in self._exception_middleware:
  97. response = middleware_method(request, e)

 ▶ Local vars

• /home/rupert/proj/metafor/metaforSVN/cmip5q/cmip5q/cmip5q/protoq/views.pyingenericDoc 68. logging.debug('ok thus far')
  68. c=cimHandler(obj)
  69. try:
  70. cmethod=getattr(c,method)
  71. except:
  72. return render_badrequest('error.html',{'message':'Method %s not known as a generic document handler'%method})
  73. logging.debug('made it')
  74. return cmethod() ...
  76. def persistedDoc(request,docType,uri,version=0):
  77. ''' persisted document handling'''
  78. if docType not in ('platform','experiment','simulation','component','datacontainer'):
  79. return HttpResponseBadRequest('Invalid document type requests - %s'%docType)
  80. set=CIMObject.objects.filter(uri=uri)

 ▶ Local vars

• /home/rupert/proj/metafor/metaforSVN/cmip5q/cmip5q/cmip5q/protoq/cimHandler.pyincimView 52. urls=commonURLs(self.obj,{'persisted':url})
  52. del urls['export']
  53. return render_to_response('cimpage.html',{'source':'Export to CMIP5','obj':self.obj,'html':html,
  54. 'urls':urls})
  56. def cimView(self):
  57. ''' View this in the CIMView interface '''
  58. html=self.obj.cimView() ...
  59. urls=commonURLs(self.obj,{})
  60. del urls['cimView'] # we've just done it
  61. return render_to_response('cimview.html',{'source':'cimView','obj':self.obj,'viewhtml':html,'urls':urls})

 ▶ Local vars

• /home/rupert/proj/metafor/metaforSVN/cmip5q/cmip5q/cmip5q/protoq/models.pyincimView 304. # schematron = ET.Schematron(sct_doc)
  304. # return schematron.validate(CIMFragment)
  306. def cimView(self):
  307. ''' All document types should be viewable in the CIM View interface '''
  308. cv=CIMViewer()
  309. self.XMLO=self.xmlobject()
  310. return cv.cimViewDoc(self.XMLO) ...
  313. def export(self):
  314. ''' Make available for export in the atom feed '''
  315. # first redo validation to make sure this really is ok
  316. if self.isComplete:

 ▶ Local vars

• /home/rupert/proj/metafor/metaforSVN/cmip5q/cmip5q/cmip5q/protoq/cimHandling.pyincimViewDoc 303. m = modelView(model, 'model')
  303. m.getnavname(model)
  304. mods.append(m)
  306. #if viewtype == 'model':
  307. #m=modelView(CIMfile, viewtype)
  308. #m=modelView(CIMdoc,'model')
  309. m=modelView(qnModels[0],'model') ...
  310. m.genhtml(qnModels[0])
  311. #return render_to_response('ModelView.html',{'mods':mods,'exps':exps,'sims':sims,'datas':datas, 'code':m.code})
  313. #cimHtml=''
  315. #return cimHtml

 ▶ Local vars

This may be a red herring but thought i'd raise a ticket in case. Currently there is a 'inputs required by this component' box on the top model level page. Is this actually necessary, given we don't ask for inputs required at this level. If it is the case that we don't need it I can add a 'if top level, don't expose' type code.

link the viewer software to the questionnaire

Go through all the dates and make them consistent with one way of doing it, and be clear about how many years of simulation are expected.

Allow the choice of grid at the higher realm levels to be filtered downwards onto it's components to avoid repetitive entry of the same information.

Copying in from email discussion:

We are discussing at GFDL how the fact that a dataset has met QC L2 will be entered into the CMIP5 system. Does the Metafor questionnaire capture QC information? Since it doesn't deal with FileCIM I imagine not but I want to check.

Is the Metafor services team working on a mechanism of entering and storing QC flags?

Steven

reply:

However, we did discuss it at the recent service meeting, and I believe it's high on Gerry's to-do list once the questionnaire is released.

I don't think it would be hard to do (to add a new interface in the questionnaire structure), and the CIM supports the concept. I doubt it'd be more than a few days work, whenever we do it.

Cheers Bryan

The id of the stc numerical requirement on experiment 5.4 needs fixed to read stc.033.

Hi Gerry

Are we going to modify the cmip5qn:qnDRS in the atom feed so that it references the DRS experiment name? 

right now the cmip5qn:qnDRS  uses the experiment long name e.g.

ccmip5qn:qnDRS="MOHC_HadGEM2-A_6.4a sstClimAerosol". 

• Which granularity shell the underlying data have? We have two (three) level: experiment and dataset and additionally something like grouped data.

a) An 'Experiment' is the 'header' of datasets collected information about all underlying datasets b) 'Dataset' is the climate data itself c) 'Grouped data' are datasets collected from at least two experiments under an new 'header'.

• In which form gets the CIM creation tool the metadata information:

a) XML b) Java properties (key=value) c) other

Met Office (Mark)
Type: Vocabulary
Location: Enumeration Lists

There are some rogue enumeration values (e.g. cats and dogs in the precipitation types). In other lists, there appear to be some values missing e.g. land surface energy balance.

Review enumeration lists before release.

I thought we had decided to do away with the idea of component 'Prognostic variables', but I see they are still part of the mindmaps. If this is something we still want to collect, then do we need a seperate panel at the relevant 'level - 1' components page to collect names of these prognostic variables? I think this would be separate to 'parameters'.

Design search UI to optimise usabilty.