Unverified Commit c1196d8f authored by David Read's avatar David Read Committed by GitHub
Browse files

Merge pull request #34 from davidread/travis-ci-setup

Add Travis CI setup
parents 4cb5c78d 163f1e3f
*.pyc
*.swp
*egg-info
.ropeproject
node_modules
bower_components
# Byte-compiled / optimized / DLL files
__pycache__/
*.py[cod]
# C extensions
*.so
# Distribution / packaging
.Python
env/
build/
develop-eggs/
dist/
sdist/
*.egg-info/
.installed.cfg
*.egg
# PyInstaller
# Usually these files are written by a python script from a template
# before PyInstaller builds the exe, so as to inject date/other infos into it.
*.manifest
*.spec
# Installer logs
pip-log.txt
pip-delete-this-directory.txt
# Unit test / coverage reports
htmlcov/
.tox/
.coverage
.cache
nosetests.xml
coverage.xml
# Sphinx documentation
docs/_build/
\ No newline at end of file
language: python
sudo: required
# use an older trusty image, because the newer images cause build errors with
# psycopg2 that comes with CKAN<2.8:
#  "Error: could not determine PostgreSQL version from '10.1'"
# see https://github.com/travis-ci/travis-ci/issues/8897
dist: trusty
group: deprecated-2017Q4
# matrix
python:
- 2.7
env:
- CKANVERSION=master
- CKANVERSION=2.7
- CKANVERSION=2.8
# tests
services:
- postgresql
- redis-server
install:
- bash bin/travis-build.bash
- pip install coveralls
script: sh bin/travis-run.sh
after_success:
- coveralls
# additional jobs
matrix:
include:
- name: "Flake8 on Python 3.7"
dist: xenial # required for Python 3.7
cache: pip
install: pip install flake8
script:
- flake8 --version
- flake8 . --count --max-complexity=10 --max-line-length=127 --statistics --exclude ckan,ckanext-hierarchy
python: 3.7
# overwrite matrix
env:
- FLAKE8=true
- CKANVERSION=master
\ No newline at end of file
GNU AFFERO GENERAL PUBLIC LICENSE
GNU AFFERO GENERAL PUBLIC LICENSE
Version 3, 19 November 2007
Copyright (C) 2007 Free Software Foundation, Inc. <http://fsf.org/>
......
recursive-include ckanext/hierarchy/public *
recursive-include ckanext/hierarchy/templates *
recursive-include ckanext/hierarchy/fanstatic *
include README.rst
include LICENSE
include requirements.txt
recursive-include ckanext/hierarchy *.html *.json *.js *.less *.css *.mo
[![Travis CI status](https://travis-ci.org/davidread/ckanext-hierarchy.svg?branch=master)](https://travis-ci.org//ckanext-hierarchy)
[![Latest version on pypi](https://img.shields.io/pypi/v/ckanext-hierarchy.svg)](https://pypi.org/project/ckanext-hierarchy/)
[![License](https://img.shields.io/pypi/l/ckanext-hierarchy.svg)](https://pypi.org/project/ckanext-hierarchy/)
# ckanext-hierarchy - Organization hierarchy for CKAN
Organizations can be arranged into a tree hierarchy.
......@@ -13,7 +21,7 @@ organization:
When viewing an organization you see its context within the tree in the side bar. In addition you can widen search of the organization's datasets to include datasets in sub-organizations too:
![Screenshot of organization page](screenshots/org_page.png)
## Detail
## Technical details
Forms (hierachy_form plugin):
* /organization/new
......@@ -63,23 +71,85 @@ the short name or acronym (more convenient for display).
TODO:
* make the trees prettier with JSTree
## Compatibility
## Requirements
This extension requires CKAN v2.7 or later.
## Installation
Install the extension in your python environment
```
$ . /usr/lib/ckan/default/bin/activate
(pyenv) $ cd /usr/lib/ckan/default/src
(pyenv) $ pip install -e "git+https://github.com/davidread/ckanext-hierarchy.git#egg=ckanext-hierarchy"
```
Then change your CKAN ini file (e.g. development.ini or production.ini). Note that hierarchy_display
To install ckanext-hierarchy:
1. Activate your CKAN virtual environment, for example:
. /usr/lib/ckan/default/bin/activate
2. Install the ckanext-hierarchy Python package into your virtual environment:
cd /usr/lib/ckan/default/src
pip install -e "git+https://github.com/davidread/ckanext-hierarchy.git#egg=ckanext-hierarchy"
3. Add ``hierarchy_display`` and ``hierarchy_form`` to the ``ckan.plugins`` setting in your CKAN
config file (by default the config file is located at
``/etc/ckan/default/production.ini``). Note that hierarchy_display
should come before hierarchy_form
```
ckan.plugins = stats text_view recline_view ... hierarchy_display hierarchy_form
```
e.g.:
ckan.plugins = stats text_view recline_view ... hierarchy_display hierarchy_form
4. Restart CKAN. For example if you've deployed CKAN with Apache on Ubuntu:
sudo service apache2 reload
## Config settings
None at present
## Tests
To run the tests, do::
nosetests --nologcapture --with-pylons=test.ini
To run the tests and produce a coverage report, first make sure you have
coverage installed in your virtualenv (``pip install coverage``) then run::
nosetests --nologcapture --with-pylons=test.ini --with-coverage --cover-package=ckanext.hierarchy --cover-inclusive --cover-erase --cover-tests
## Releasing a new version of ckanext-hierarchy
ckanext-hierarchy should be available on PyPI as https://pypi.org/project/ckanext-hierarchy.
To publish a new version to PyPI follow these steps:
1. Update the version number in the ``setup.py`` file.
See `PEP 440 <http://legacy.python.org/dev/peps/pep-0440/#public-version-identifiers>`_
for how to choose version numbers.
2. Make sure you have the latest version of necessary packages:
pip install --upgrade setuptools wheel twine
3. Create a source and binary distributions of the new version:
python setup.py sdist bdist_wheel && twine check dist/*
Fix any errors you get.
4. Upload the source distribution to PyPI:
twine upload dist/*
5. Commit any outstanding changes:
git commit -a
6. Tag the new release of the project on GitHub with the version number from
the ``setup.py`` file. For example if the version number in ``setup.py`` is
0.0.1 then do:
git tag 0.0.1
git push --tags
## Licence and copyright
......
#!/bin/bash
set -e
echo "This is travis-build.bash..."
echo "Installing the packages that CKAN requires..."
sudo apt-get update -qq
sudo apt-get install solr-jetty
echo "Installing CKAN and its Python dependencies..."
git clone https://github.com/ckan/ckan
cd ckan
if [ $CKANVERSION == 'master' ]
then
echo "CKAN version: master"
else
CKAN_TAG=$(git tag | grep ^ckan-$CKANVERSION | sort --version-sort | tail -n 1)
git checkout $CKAN_TAG
echo "CKAN version: ${CKAN_TAG#ckan-}"
fi
# install the recommended version of setuptools
if [ -f requirement-setuptools.txt ]
then
echo "Updating setuptools..."
pip install -r requirement-setuptools.txt
fi
if [ $CKANVERSION == '2.7' ]
then
echo "Installing setuptools"
pip install setuptools==39.0.1
fi
python setup.py develop
pip install -r requirements.txt
pip install -r dev-requirements.txt
cd -
echo "Creating the PostgreSQL user and database..."
sudo -u postgres psql -c "CREATE USER ckan_default WITH PASSWORD 'pass';"
sudo -u postgres psql -c 'CREATE DATABASE ckan_test WITH OWNER ckan_default;'
echo "Setting up Solr..."
# Solr is multicore for tests on ckan master, but it's easier to run tests on
# Travis single-core. See https://github.com/ckan/ckan/issues/2972
sed -i -e 's/solr_url.*/solr_url = http:\/\/127.0.0.1:8983\/solr/' ckan/test-core.ini
printf "NO_START=0\nJETTY_HOST=127.0.0.1\nJETTY_PORT=8983\nJAVA_HOME=$JAVA_HOME" | sudo tee /etc/default/jetty
sudo cp ckan/ckan/config/solr/schema.xml /etc/solr/conf/schema.xml
sudo service jetty restart
echo "Initialising the database..."
cd ckan
paster db init -c test-core.ini
cd -
echo "Installing ckanext-hierarchy and its requirements..."
python setup.py develop
pip install -r dev-requirements.txt
echo "Moving test.ini into a subdir..."
mkdir subdir
mv test.ini subdir
echo "travis-build.bash is done."
\ No newline at end of file
#!/bin/sh -e
set -ex
flake8 --version
# stop the build if there are Python syntax errors or undefined names
flake8 . --count --select=E901,E999,F821,F822,F823 --show-source --statistics --exclude ckan,ckanext-hierarchy
nosetests --ckan \
--nologcapture \
--with-pylons=subdir/test.ini \
--with-coverage \
--cover-package=ckanext.hierarchy \
--cover-inclusive \
--cover-erase \
--cover-tests
# strict linting
flake8 . --count --max-complexity=10 --max-line-length=127 --statistics --exclude ckan,ckanext-hierarchy
\ No newline at end of file
# encoding: utf-8
# this is a namespace package
try:
import pkg_resources
......
# encoding: utf-8
# this is a namespace package
try:
import pkg_resources
......
import re
import ckan.plugins as p
import ckan.model as model
from ckan.common import request
def group_tree(organizations=[], type_='organization'):
full_tree_list = p.toolkit.get_action('group_tree')({}, {'type': type_})
......@@ -15,9 +14,9 @@ def group_tree(organizations=[], type_='organization'):
def group_tree_filter(organizations, group_tree_list, highlight=False):
# this method leaves only the sections of the tree corresponding to the list
# since it was developed for the users, all children organizations from the
# organizations in the list are included
# this method leaves only the sections of the tree corresponding to the
# list since it was developed for the users, all children organizations
# from the organizations in the list are included
def traverse_select_highlighted(group_tree, selection=[], highlight=False):
# add highlighted branches to the filtered tree
if group_tree['highlighted']:
......@@ -31,7 +30,7 @@ def group_tree_filter(organizations, group_tree_list, highlight=False):
for child in group_tree.get('children', []):
traverse_select_highlighted(child, selection)
filtered_tree=[]
filtered_tree = []
# first highlights all the organizations from the list in the three
for group in group_tree_highlight(organizations, group_tree_list):
traverse_select_highlighted(group, filtered_tree, highlight)
......@@ -39,28 +38,34 @@ def group_tree_filter(organizations, group_tree_list, highlight=False):
return filtered_tree
def group_tree_section(id_, type_='organization', include_parents=True, include_siblings=True):
def group_tree_section(id_, type_='organization', include_parents=True,
include_siblings=True):
return p.toolkit.get_action('group_tree_section')(
{'include_parents':include_parents, 'include_siblings':include_siblings}, {'id': id_, 'type': type_,})
{'include_parents': include_parents,
'include_siblings': include_siblings},
{'id': id_, 'type': type_, })
def group_tree_parents(id_, type_='organization'):
tree_node = p.toolkit.get_action('organization_show')({},{'id':id_})
if (tree_node['groups']):
parent_id = tree_node['groups'][0]['name']
parent_node = p.toolkit.get_action('organization_show')({},{'id':parent_id})
return group_tree_parents(parent_id) + [parent_node]
else:
return []
tree_node = p.toolkit.get_action('organization_show')({}, {'id': id_})
if (tree_node['groups']):
parent_id = tree_node['groups'][0]['name']
parent_node = \
p.toolkit.get_action('organization_show')({}, {'id': parent_id})
return group_tree_parents(parent_id) + [parent_node]
else:
return []
def group_tree_get_longname(id_, default="", type_='organization'):
tree_node = p.toolkit.get_action('organization_show')({},{'id':id_})
longname = tree_node.get("longname", default)
if not longname:
return default
return longname
tree_node = p.toolkit.get_action('organization_show')({}, {'id': id_})
longname = tree_node.get("longname", default)
if not longname:
return default
return longname
def group_tree_highlight(organizations, group_tree_list):
def group_tree_highlight(organizations, group_tree_list):
def traverse_highlight(group_tree, name_list):
if group_tree.get('name', "") in name_list:
group_tree['highlighted'] = True
......@@ -69,13 +74,14 @@ def group_tree_highlight(organizations, group_tree_list):
for child in group_tree.get('children', []):
traverse_highlight(child, name_list)
selected_names = [ o.get('name',None) for o in organizations]
selected_names = [o.get('name', None) for o in organizations]
print(selected_names)
for group in group_tree_list:
traverse_highlight(group, selected_names)
return group_tree_list
def get_allowable_parent_groups(group_id):
if group_id:
group = model.Group.get(group_id)
......@@ -86,6 +92,7 @@ def get_allowable_parent_groups(group_id):
group_type='organization')
return allowable_parent_groups
def is_include_children_selected(fields):
include_children_selected = False
if request.params.get('include_children'):
......
......@@ -38,24 +38,28 @@ def group_tree_section(context, data_dict):
group_type = data_dict.get('type', 'group')
if group.type != group_type:
how_type_was_set = 'was specified' if data_dict.get('type') \
else 'is filtered by default'
else 'is filtered by default'
raise p.toolkit.ValidationError(
'Group type is "%s" not "%s" that %s' %
(group.type, group_type, how_type_was_set))
include_parents = context.get('include_parents', True)
include_siblings = context.get('include_siblings', True)
if include_parents:
root_group = (group.get_parent_group_hierarchy(type=group_type) or [group])[0]
root_group = (group.get_parent_group_hierarchy(type=group_type)
or [group])[0]
else:
root_group = group
if include_siblings or root_group==group:
if include_siblings or root_group == group:
return _group_tree_branch(root_group, highlight_group_name=group.name,
type=group_type)
else:
section_subtree = _group_tree_branch(group, highlight_group_name=group.name,
section_subtree = _group_tree_branch(group,
highlight_group_name=group.name,
type=group_type)
return _nest_group_tree_list(group.get_parent_group_hierarchy(type=group_type),
section_subtree)
return _nest_group_tree_list(
group.get_parent_group_hierarchy(type=group_type),
section_subtree)
def _nest_group_tree_list(group_tree_list, group_tree_leaf):
'''Returns a tree branch composed by nesting the groups in the list.
......@@ -69,9 +73,9 @@ def _nest_group_tree_list(group_tree_list, group_tree_leaf):
for group in group_tree_list:
log.debug(group)
node = GroupTreeNode(
{'id': group.id,
'name': group.name,
'title': group.title})
{'id': group.id,
'name': group.name,
'title': group.title})
if not root_node:
root_node = last_node = node
else:
......
import logging
import re
import ckan.plugins as p
from ckan import model
from ckan.lib.plugins import DefaultOrganizationForm
from ckan.lib.plugins import DefaultGroupForm
import ckan.logic.schema as s
from ckan.common import c, request
from ckan.common import c
from ckanext.hierarchy.logic import action
from ckanext.hierarchy import helpers
......@@ -16,8 +13,8 @@ log = logging.getLogger(__name__)
# This plugin is designed to work only these versions of CKAN
p.toolkit.check_ckan_version(min_version='2.0')
def custom_convert_from_extras(key, data, errors, context):
def custom_convert_from_extras(key, data, errors, context):
'''Converts values from extras, tailored for groups.'''
# Set to empty string to remove Missing objects
......@@ -27,18 +24,18 @@ def custom_convert_from_extras(key, data, errors, context):
for data_key in data.keys():
if (data_key[0] == 'extras'):
data_value = data[data_key]
if( 'key' in data_value and data_value['key'] == key[-1]):
data[key] = data_value['value']
to_remove.append(data_key)
break
if 'key' in data_value and data_value['key'] == key[-1]:
data[key] = data_value['value']
to_remove.append(data_key)
break
else:
return
for remove_key in to_remove:
del data[remove_key]
class HierarchyDisplay(p.SingletonPlugin):
class HierarchyDisplay(p.SingletonPlugin):
p.implements(p.IConfigurer, inherit=True)
p.implements(p.IActions, inherit=True)
p.implements(p.ITemplateHelpers, inherit=True)
......@@ -66,11 +63,12 @@ class HierarchyDisplay(p.SingletonPlugin):
'group_tree_parents': helpers.group_tree_parents,
'group_tree_get_longname': helpers.group_tree_get_longname,
'group_tree_highlight': helpers.group_tree_highlight,
'get_allowable_parent_groups': helpers.get_allowable_parent_groups,
'is_include_children_selected': helpers.is_include_children_selected,
'get_allowable_parent_groups':
helpers.get_allowable_parent_groups,
'is_include_children_selected':
helpers.is_include_children_selected,
}
# IPackageController
def before_search(self, search_params):
......@@ -95,9 +93,9 @@ class HierarchyDisplay(p.SingletonPlugin):
# Remove the param from the fields - NB no longer works
# e.g. [('include_children', 'True')]
new_fields = set()
for field,value in c.fields:
for field, value in c.fields:
if (field != 'include_children'):
new_fields.add((field,value))
new_fields.add((field, value))
c.fields = list(new_fields)
# parse the query string to check if children are requested
......@@ -128,7 +126,7 @@ class HierarchyDisplay(p.SingletonPlugin):
' OR '.join(
'organization:{}'.format(org_name)
for org_name in [c.group_dict.get('name')] +
children_names))
children_names))
search_params['q'] = query
# add it back to fields
# c.fields += [('include_children', 'True')]
......@@ -140,10 +138,8 @@ class HierarchyDisplay(p.SingletonPlugin):
class HierarchyForm(p.SingletonPlugin, DefaultOrganizationForm):
p.implements(p.IGroupForm, inherit=True)
# IGroupForm
def group_types(self):
......@@ -156,4 +152,5 @@ class HierarchyForm(p.SingletonPlugin, DefaultOrganizationForm):
from pylons import tmpl_context as c
group_id = data_dict.get('id')
c.allowable_parent_groups = helpers.get_allowable_parent_groups(group_id)
c.allowable_parent_groups = \
helpers.get_allowable_parent_groups(group_id)
......@@ -2,8 +2,6 @@ import nose.tools
from bs4 import BeautifulSoup
from ckan.tests import helpers, factories
from ckan.lib import search
from ckan.common import c
from ckan import model
......@@ -54,31 +52,14 @@ class TestSearchApi(object):
def create_fixtures():
parent_org = factories.Organization(name='parent_org', title='Parent')
child_org = factories.Organization(name='child_org', title='child')
member = model.Member(group=model.Group.get(child_org['id']), table_id=parent_org['id'], table_name='group', capacity='parent')
member = model.Member(
group=model.Group.get(child_org['id']),
table_id=parent_org['id'], table_name='group', capacity='parent')
model.Session.add(member)
model.Session.commit()
parent_dataset = factories.Dataset(name='parent', title='Parent',
owner_org=parent_org['id'])
owner_org=parent_org['id'])
child_dataset = factories.Dataset(name='child', title='Child',
owner_org=child_org['id'])
decoy_dataset = factories.Dataset(name='decoy', title='Decoy')
owner_org=child_org['id'])
factories.Dataset(name='decoy', title='Decoy')
return parent_org, child_org, parent_dataset, child_dataset
class LoadPluginsBase(object):
@classmethod
def setup_class(cls):
'''Load plugins'''
import ckan.plugins as p
try:
config['ckan.plugins'] = ' '.join(cls._load_plugins)
except AttributeError:
pass
cls._get_test_app()
@classmethod
def teardown_class(cls):
import ckan.plugins as p
for plugin in reversed(getattr(cls, '_load_plugins', [])):
p.unload(plugin)
flake8 # for the travis build
\ No newline at end of file