compare merge configs across envs

Counting on a long list of "click here, type this, then click that..." to make changes in the UI has been the source of subtle inconsistencies in other applications in the past, leading to Production not actually running the thing that was built and tested in non-prod... Trying to avoid that and bake good habits into our workflow as we get up to speed with Grouper and deploy it in our Production environment.

Bruce Timberlake 3 days ago
It feels sort of like https://spaces.at.internet2.edu/display/Grouper/Import-Export could be a tool to use, but I'm not sure if that's what others do or would recommend, etc.

Bruce Timberlake 3 days ago
It feels like the .properties files and overlays would be the best way to manage changes, and is how we handle all of our other applications. But the docs say
In a patch to Grouper 2.4, and in Grouper 2.5+, Grouper allows configuration to be stored in the database rather than in configuration files. This is the recommended approach.
(edited)

Chris Hyzer 3 days ago
whats an example of something you are promoting? or a few examples?

Chris Hyzer 3 days ago
you can export a config file in the configuration screen, copy the part you want, and import that in new env (with configuration screen). do you have more things to migrate e.g. folders or groups? maybe a template would help or the grouper object sync
https://spaces.at.internet2.edu/display/Grouper/Syncing+objects+to+Grouper+from+SQL

Bruce Timberlake 3 days ago
I don't have anything specific yet, since we are just bringing up our first pilot user/application. It's currently in our Sandbox environment but we will have the other VMs prepared shortly. All my experience has been with templated properties files that get filled in with Ansible vars. We are trying to avoid point-click in the UI at all costs if possible, as that is not always repeatable depending on who does the work, etc. I can chat with Liam more to see if we can define it better, but I guess I was looking for the overall "philosophy" of managing configs through multiple environments... with ideas/examples of how people are doing it now in their own Production environments.

Jonathan Keller 3 days ago
We are in the same pilot phase - but I have had a number of the same concerns. At least for establishing the base structure of the servers, I've been creating GSH scripts which we can then promote to each environment as we move forward.
I created simple wrappers around the base GSH commands to make group addition one-liners. And I'm triggering the main GDG template from GSH and plan to do the same for creating the application level trees so we can deploy those across environments.

Jonathan Keller 3 days ago
It will probably be some manual setup in dev to practice, then delete the tree and script it up for replay there and in the testing and production environments.

Jonathan Keller 3 days ago
guh - this is what happens when your POC goes on too long. I apparently had written a GSH scriptable use of the "service" template.

Bruce Timberlake 3 days ago
I have to finish building up all of our environments so I can actually start using the app a bit. Liam's been doing all that work for us so far; a lot of this is conceptual-only knowledge for me at this point :slightly_smiling_face:

Bruce Timberlake 3 days ago
My main concern with gsh is that it gets run inside the containers, right, through an interactive shell? In addition to avoiding humans doing pointy-clicky in the UI for config management, I'm trying to avoid ever typing commands into an interactive shell on a container.
External, repeatable interactions invoked by the orchestration/config management tool of your choice (Ansible, Puppet, etc) is very much the preferred way for us if at all possible... I need to do some more reading I guess, to see if those scripts can be invoked externally somehow.
:100:
1

Carey Black 3 days ago
A lovely question. Well asked and full of possible opinionated answers to be given. :slightly_smiling_face:
I am not convinced that the “goal” of having the configs “match” is a good goal. There are going to at least be “value differences”. ( DB names, usernames, passwords, etc…)
And if you name things with an “ENV” prefix/suffix then those are really important and challenging to manage too.
However, I have actually been thinking about possibly using a config pattern that could keep the “main config” identical and allow for values to be environment specific too.
Config values set like this:
BLA_VALUE=<config_value for the ENV>
Grouper properties set this way:
bla.bla.bla = $$BLA_VALUE$$
REF: https://spaces.at.internet2.edu/display/Grouper/Grouper+configuration+files+and+overlays
Section “Refer to other properties in that config file”

Bruce Timberlake 3 days ago
Yes, definitely the environment specific difference need to be accounted for. Dev and Prod will never run in an "identical" configuration. But currently those differences are managed via the "docker run" script with a pile of "-e FOO=bar" lines :slightly_smiling_face: And opinions/options are most welcome!

Bruce Timberlake 3 days ago
And I'm not opposed to lots of -e switches, as I can manage that script and its contents with an Ansible/Jinja template.

Carey Black 3 days ago
Basically If the “Grouper properties” could all be in files. ( and identical )
Then that could all be source controlled and promoted with container release.
And the “Config values” could be imported with the UI so that the local env values are all supplied with $$sub_this_value$$ approaches.

Bruce Timberlake 3 days ago
We've been reading about the overlay files, but then there's that darned "Database is the recommended option" statement...

Carey Black 3 days ago
The approach that I am considering is to put the values in the UI, but the “structure” in the files.

Bruce Timberlake 3 days ago
I'm just super-leery of needing to do lots of point-click in the UI... that's just too error-prone in my experience and opinion... if it must be done, it must be. But I would love to just deal with .properties files that I can lay down with Ansible + Jinja templates so that all the config options are in a repository someplace, version controlled, and can be validated in "tripwire" mode at any time so you know what's running in a given environment.

Carey Black 3 days ago
And as far as the question of “GSH remote execution”…
GSH templates could be created.
And called from WS calls.
To add/change values on the server config. :slightly_smiling_face:

Bruce Timberlake 3 days ago
We've still got so far to go with Grouper, I may be asking this question too early. But we want to head off dangerous/dumb decisions early before they get baked into our systems and processes :slightly_smiling_face:

Jonathan Keller 3 days ago
We have not even attempted the automation step, but we would have the ability (perhaps via ansible) to ssh into the UI server during deployments to run incremental scripts. We do this on our databases now with Liquibase and its worked well for us.

Carey Black 3 days ago
RE: “to run incremental scripts”
I have considered the idea of a “lifecycle hook” to read a directory and “run scripts” from the container on startup too.
REF: https://spaces.at.internet2.edu/display/Grouper/Hooks

Carey Black 3 days ago
With the move of the project to “builder style commands” it makes that approach a lot more … approachable.

Carey Black 3 days ago
Likely would delay startup times… but that is the trade off.. “do lots of checks/creates” or “just start”.

Bruce Timberlake 3 days ago
Yeah, I'm fine with delayed startup times (within reason)... this looks like an interesting approach though.

Carey Black 3 days ago
However, a really fancy programmer could also use a kind of “version marker” in the scripts/directory to know if they need run the second time too. :slightly_smiling_face:

Jonathan Keller 3 days ago
tools like liquibase keep a log of all previously run sets - if we had a unique identifier attached to each script - then the checks could run fairly quickly

Bruce Timberlake 3 days ago
Thanks for the lead! I'm signing off for the day, so thanks to all for the info so far, and I look forward to hearing more about this in the future too!

Jonathan Keller 3 days ago
actually - doesn't grouper do that already for the DDL updates?

Carey Black 3 days ago
Yea.. but that is all “internal stuff”….. And your scripts likely would need its’ own “high water mark”…. that could go up before the grouper DDL marker would change.

Jonathan Keller 3 days ago
I just meant that the concept was already there within the grouper code - I would have expected this to be a separate layer

Carey Black 3 days ago
:shrug: internal implementation detail.
Not for use by “users”. :slightly_smiling_face:
But the idea is straight forward. :slightly_smiling_face:

Jonathan Keller 3 days ago
if I wanted to look into this, would the hooksInit() lifecycle hook be late enough in the process to attempt to run a GSH script?

Carey Black 3 days ago
I think that kind of depends on “what your doing”.
Some cases might need to be in “ddlInit” ( but likely none ),
Others are either in
grouperStartup, or hooksInit .
I think.. hooksInit is “the last one”… ( from memory )

Jonathan Keller 3 days ago
I looked at the other 3 - they seemed like they would be too early in the startup process from their name and comments.

Jonathan Keller 3 days ago
I was thinking of causing a GSH environment to spawn - however that can be done within a given server - then read in the scripts from a path - confirm which ones need to be run, and exec the new ones

Carey Black 3 days ago
well… to early or “just right”.
If you want to do some custom “DDL” (for grouper to use: Think hibernate customizations/additions) you might need the ddlInit phase.
However.. if you want to use GSH ( specifically, instead of just using the Grouper API with a config file of inputs…) then yea.. the JDBC connection needs to already be open and usable. ( AFAIK: hooksInit )

Carey Black 3 days ago
However there is a point where you are low enough in the system that GSH is an abstraction that costs you something to use too.

Jonathan Keller 3 days ago
I've noticed the overhead in starting GSH - but I would like for these to be scripts which can be tested manually, and then packaged up in the docker image for deployment

Carey Black 3 days ago
Meaning, your Hook could read your config file and “call java or JDBC, or anything you want it to do”. :slightly_smiling_face:
Yea. startup time ( per script ) would add up if you have N ( dozen ) to run every startup.

Jonathan Keller 3 days ago
I would likely make the checks for the files needing to be run be in the hook itself - only if there was something new would it instantiate a GSH environment
:+1:
1

Jonathan Keller 3 days ago
I haven't tried hacking grouper code directly - I'm going to have to look at the docs on how to do that.

Carey Black 3 days ago
Still running N would takes N*X seconds of startup time to add to the container.

Jonathan Keller 3 days ago
right - but if there were N scripts which contained configuration that we expected to be there on next startup - that would be ok. I think I would only have this run on the UI server...because of the need to access the templates. I've had problems with that on the non-UI servers

Jonathan Keller 3 days ago
not sure what to do about clustering...we were only going to have one UI server initially...bleh

Carey Black 3 days ago
GSH is a thin wrapper ( in groovy ) around the Java API. And the Java API would only be usable after the full hibernate connections were established too.

Carey Black 3 days ago
The “state” would need to be in the DB. ( As all good state should be. :slightly_smiling_face: ) (edited)

Jonathan Keller 3 days ago
yea - and need some sort of semaphore there to prevent scripts from running double - as the quartz scheduler handles for the daemon

Carey Black 3 days ago
maybe best done that way…
Or maybe the orchestration layer could play a role there?

Jonathan Keller 3 days ago
that I know nothing about...I think I've seen references in the docs

Carey Black 3 days ago
IMHO. Orchestration becomes another possible failure mode for the system.
If you bake it into the code/process (semephore) then the code/process should NEVER fail.
You would need to decide if the semephore is a blocking function for any other things starting up (or needing to be restarted after?) and what to do if that is those cases…
But if you plan your approach to be “safe to re-run” and “forward and backwards” compatible… It should all be good. :slightly_smiling_face:

Carey Black 3 days ago
You might even choose to use the Grouper internal messaging system too. :slightly_smiling_face:

Carey Black 3 days ago
And for a staring point to see how thin the GSH wrapper is… anything in that directory is a reasonable place to look.
https://github.com/Internet2/grouper/blob/master/grouper/src/grouper/edu/internet2/middleware/grouper/app/gsh/addComposite.java

Jonathan Keller 3 days ago
thanks for the info! I hope I have time to look into this

Chris Bongaarts (UMN) 3 days ago
you can also choose to have grouper NOT look in the database for configs, and rely exclusively on files, like in the good ole days. DB is recommended but the old way is still supported.

Chris Bongaarts (UMN) 3 days ago
(last i checked)
:yes:
1

Chris Hyzer 2 days ago
One goal of the UI wizards is to have a much more friendly interface with documentation and validation to configure Grouper. Using properties files has proven to be error prone, time consuming, confusing, and frustrating. If there are more features to add to the DB config to make it more palatable we are open to that. It seems like people want a checkbox to identify the config as environment specific as opposed to institution specific (e.g. prod vs test). If we add metadata like this then the export/import will probably need to be JSON...

Lacey Vickery 2 days ago
We went down this road last year, thinking we needed to keep everything in config overlays using env variables per instance and track changes in git. Since Grouper now supports config history and passing most configs that differ per instance as env variables to containers, I’ve ditched the overlays and just rely on docker env variables and config history if I need to roll something back. We do use GSH scripts to build out loaders/ref/basis groups so it’s repeatable across different environments. I don’t know that it’s worth the headache to try to automate all aspects of running grouper, it seems like there will inevitably be a point when you just need to do something in the UI (building reports for example) which makes sense from a distributed access management perspective.

Carey Black 2 days ago
RE: “If there are more features to add to the DB config to make it more palatable we are open to that.”
I think this would be helpful.
When an import of a config file ( via UI or API [Can that be done?] ) is done, then all of those changes are grouped/named with a single config “change/commit”. So that they could all be removed as a single unit/action.
While also preserving the ability to individually change single lines too.
NOTE1: Changing a single line could exclude it from the “config change/commit” rollback too. ( Very hard to deal with 3 way merge logic. :disappointed: )
OR
NOTE2: Or rolling back a would “just revert all settings in the set” regardless of other changes that might have been done after/outside of the “change/commit” too.
NOTE1 or NOTE2 : Projects choice. Document it and the deployer should be able to understand and operate accordingly. (edited)

Carey Black 2 days ago
@Lacey Vickery. I have even considered trying to use a ENV config value that would operate as a Prefix for other $$CONFIG_VALUE$$ logic. Though I have not yet validate that the below would work “as is”.. it might take a static method to do the last lookup…
The advantages are
The configs would be 100% together. ( All configs, for all envs )
That allows you to see where deviations exist too.
It could be file based. ( or programmatically and blindly loaded to an instance during a deploy event. )
Disadvantages a chain of substitutions to look through to see errors.
SAMPLE / Example of the idea:
ENV = PROD

1. ENV = QA
2. E NV = DEV
3. config value mappings
   PROD.GRANT_ALL_OPTIN= false
   QA.GRANT_ALL_OPTIN = false
   DEV.GRANT_ALL_OPTIN = true
4. grouper config with ENV specific values used from above
   grouper.env.name = $$ENV$$
   ….
   groups.create.grant.all.optin = ${GrouperConfig.retrieveConfig().propertyValueString(“$$ENV$$.$$GRANT_ALL_OPTIN$$“)
5. NOTE: GrouperConfig would only work for grouper.properties. Other classes would be needed for other files. But I think they all exist in the code base.
6. NOTE: I am not sure that the code currently would work with two $$ revs on the same line as each other. It may not resolve them properly as above. However, a static method that could be “ENV” aware could also be created.
7. AKA:
8. ${MyGrouperConfigFromEnv(“GRANT_ALL_OPTIN”)}
   #
9. which would:
10. propName = GrouperConfig.retrieveConfig().propertyValueString(“ENV”) + “.” + GrouperConfig.retrieveConfig().propertyValueString(“GRANT_ALL_OPTIN”)
11. then
12. return GrouperConfig.retrieveConfig().propertyValueString(propName)
    :shrug: just have not worked that hard at it yet. :slightly_smiling_face: ( “Other bigger fish to fry.” :slightly_smiling_face: )

Chris Bongaarts (UMN) 2 days ago
in the spirit of devops (since grouper is all containery now), config should be accessbile and maintainable as code. config files lend themselves to that well, db less so (but with import/export, is still eminently doable). whether version control, "commits", etc. are internal to grouper or external (one may want to keep one's grouper configs alongside their other app configs) should be a deployment choice (or I'd argue, the latter should be preferred so grouper can focus on being grouper and not Yet Another VCS)

Chris Bongaarts (UMN) 2 days ago
as for carey's idea above, a little syntactic sugar/macros/helper methods to simplify those kind of operations would be nice

Lacey Vickery 2 days ago
@black.123 That’s an interesting approach, I hadn’t considered using an env prefix to set values conditionally. I’ve migrated our configs to the DB and am not looking back now, lol. We don’t have a very mature docker/orchestration strategy so mounting configs wasn’t a great approach. At one point I was doing something sort of similar (like below), but it was difficult to keep up with all the config differences that way and I think not all classes supported this.
ws.authn.ldap.findUserBase = ${“DC=its,DC=” + java.lang.System.getenv().get(‘GROUPER_LDAP_HOST’) + “,DC=uncc,DC=edu”}

Carey Black 2 days ago
I think that using a “.elConfig” suffix always works. ( Across all of the grouper property hierarchy files… )
So you might need to tell the config reader that the value is a “JEXL expression” that way.
And frankly I kind of like that explicit “This is a script. It is ok to run it like a script.” specification in the config anyways.
Personal choice of “configuration over convention”. :slightly_smiling_face:
:+1:
1

Chris Hyzer 2 days ago
there is still the wizard issue, if you edit with wizard and you have elconfigs, then after hitting save, things revert back since elconfig takes precendence I think? I guess you just need to remember that. maybe it would help if there were a compare/merge configs across envs in UI :slightly_smiling_face:
New

Lacey Vickery 2 hours ago
I think a compare/merge configs would be great :wink: