Re: [xwiki-devs] Xwiki Helm HA

On 23-Jul-2019, at 12:30 PM, Ashish Sharma <sharma.ashish960(a)gmail.com>

Hey Shubham, Yes I did that and it did not work, once I scale it by starting it from

The website no longer loads. On Sat, 20 Jul 2019 at 10:00 AM, Shubham Jain <jainshubham78(a)gmail.com> wrote: > That’s interesting. > > Were you able to figure out the problems when you scale up from 1 Pod?

> not, share the logs. Let’s try to figure out problem. > > Best, > Shubham. > >> On 20-Jul-2019, at 6:36 AM, Ashish Sharma <sharma.ashish960(a)gmail.com> > wrote: >> >> Hey Shubham, >> >> I have tried initialising with 1 pod and scaling it up to multiple

>> also tried session affinity, and some other cases but none of them > worked. >> >> Thanks, >> Ashish >> On Fri, 19 Jul 2019 at 4:51 PM, Shubham Jain <jainshubham78(a)gmail.com> >> wrote: >> >>> Hey Ashish, >>> >>> I don’t understand XWiki well enough to fully understand the > implications >>> of running a clustered setup by sharing file system. >>> >>> Can you try keeping the deployment size initially 1, and then later

> to >>> increase the size? Does that also cause problems? >>> >>> Best, >>> Shubham. >>> >>>> On 19-Jul-2019, at 4:03 PM, Ashish Sharma <

>>> wrote: >>>> >>>> Hey Shubham, >>>> >>>> I have moved the solr folder as exposed by the docker container to

>>> file >>>> system. I think in-memory cache is creating problem. >>>> >>>> I started 3 container of xwiki with shared file storage, but all

>>> were >>>> trying to initiate the postgress database, that is causing problem. >>>> >>>> I want to know few things about xwiki:- >>>> >>>> 1. Do we use java threads ? -> if we do then filesystem locking

>>> have >>>> been implemented for that? >>>> >>>> 2. Other stateful things in our application like caching or other >>>> filestorage that should be externalise? >>>> >>>> >>>> Thanks, >>>> Ashish >>>> >>>> On Fri, 19 Jul 2019 at 3:59 AM, Shubham Jain <

>>>> wrote: >>>> >>>>> Hey Ashish, >>>>> >>>>> I think for now you could stick to deployment. Have you been ever to >>>>> discover the data you need to move to the shared file system? Is the >>>>> in-memory caching mechanism causing problems? >>>>> >>>>> @Thomas/Vincent - Can you please help us understand the implications > of >>>>> running XWiki in a clustered mode by sharing data volumes? >>>>> >>>>> Best, >>>>> Shubham. >>>>> >>>>> >>>>> Best, >>>>> Shubham? >>>>> >>>>>> On 19-Jul-2019, at 5:35 AM, Ashish Sharma < > sharma.ashish960(a)gmail.com&gt; >>>>> wrote: >>>>>> >>>>>> Resent the email at the mail thread. >>>>>> >>>>>> On Fri, Jul 19, 2019 at 2:01 AM Ashish Sharma < >>>>> sharma.ashish960(a)gmail.com&gt; >>>>>> wrote: >>>>>> >>>>>>> Hello everyone, >>>>>>> >>>>>>> I am trying to implement xwiki helm chart in High availability but >>>>> facing >>>>>>> some problems. >>>>>>> >>>>>>> How we can implement High Availability(HA) in Kubernetes:- >>>>>>> >>>>>>> 1. Through Deployment:- In this approach we would use a shared

>>>>>>> storage, that would get attached to multiple pods(or containers) >>>>>>> 2. Through StatefulSet:- In this approach every pod(or container) >>> would >>>>>>> have it’s own different volume and state, which should be > synchronised >>>>> by >>>>>>> an consensus algorithm >>>>>>> >>>>>>> Problem faced in xwiki case:- >>>>>>> >>>>>>> 1. In deployment case we need a locking mechanism for our > filesystem, >>>>>>> moreover caching and states should also be maintained outside the >>>>> container. >>>>>>> 2. In StatefulSet the consensus algorithm is not present so it is >>>>>>> currently not possible. >>>>>>> >>>>>>> Could anyone help me through it. >>>>>>> >>>>>>> Thanks, >>>>>>> Ashish >>>>>>> >>>>>>> On Wed, 10 Jul 2019 at 10:32 AM, Shubham Jain < >>> jainshubham78(a)gmail.com&gt; >>>>>>> wrote: >>>>>>> >>>>>>>> Hey Ashish, >>>>>>>> >>>>>>>> I think Vincent/Thomas can help with the PR on the xwiki-docker >>>>> project. >>>>>>>> >>>>>>>> Regarding adding docs for High-Availability setup using shared

>>>>>>>> volumes, I think you can use could create a docs folder in the > GitHub >>>>> repo, >>>>>>>> and add the docs in Markdown format. I think you can also

> it >>>>> in >>>>>>>> XWiki blog <https://www.xwiki.org/xwiki/bin/view/Blog/>. The

>>>>>>>> should support enabling HA, and then passing a storage class

>>>>> supports >>>>>>>> shareable volumes to Pods, and then you can document examples for >>> using >>>>>>>> this with something like Rook+GKE setup. >>>>>>>> >>>>>>>> Best, >>>>>>>> Shubham. >>>>>>>> >>>>>>>> >>>>>>>>> On 08-Jul-2019, at 1:57 PM, Ashish Sharma < >>> sharma.ashish960(a)gmail.com >>>>>> >>>>>>>> wrote: >>>>>>>>> >>>>>>>>> Hello everyone, >>>>>>>>> >>>>>>>>> I require the xwiki solr docker container to do progress with my >>>>> work. I >>>>>>>>> have made a pull request >>>>>>>>> <https://github.com/xwiki-contrib/docker-xwiki/pull/22>, for

>>>>> same. >>>>>>>>> Please look into it. >>>>>>>>> >>>>>>>>> Moreover, I was working with Rook(Ceph Operator) for a shared

>>>>>>>> system >>>>>>>>> which would be a better approach to deploy our app on

>>> So I >>>>>>>>> think we should document it somewhere. Can anyone help me where >>> should >>>>>>>> I do >>>>>>>>> that. It may contain the details to configure rook-ceph, and

>>>>>>>>> connecting it. >>>>>>>>> >>>>>>>>> P.S. I am writing a blog at Medium on how to configure rook on >>> Google >>>>>>>>> Kubernetes Engine. We could also use that? >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> Ashish >>>>>>>>> >>>>>>>>> On Mon, Jun 24, 2019 at 8:31 PM Ashish Sharma < >>>>>>>> sharma.ashish960(a)gmail.com&gt; >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Hi Shubham, >>>>>>>>>> >>>>>>>>>> Thank you Shubham, I will start working on adding test cases. >>>>>>>>>> >>>>>>>>>> Thanks, >>>>>>>>>> Ashish >>>>>>>>>> >>>>>>>>>> On Mon, 24 Jun 2019 at 3:32 PM, Shubham Jain < >>>>> jainshubham78(a)gmail.com&gt; >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> Hey Ashish, >>>>>>>>>>> >>>>>>>>>>> Thanks for the update. >>>>>>>>>>> >>>>>>>>>>> Adding our chart to incubator section of helm community charts >>>>>>>> repository >>>>>>>>>>> sounds great. Although (unlike Dockerhub) this repo is for >>> community >>>>>>>> charts >>>>>>>>>>> and not really official charts of the respective

>>> but >>>>>>>>>>> supporting this chart in the community chart repo should help > with >>>>> the >>>>>>>>>>> visibility. >>>>>>>>>>> >>>>>>>>>>> Lets also work on adding tests to the helm chart to ensure > things >>>>> are >>>>>>>>>>> going fine. >>>>>>>>>>> >>>>>>>>>>> Best, >>>>>>>>>>> Shubham. >>>>>>>>>>> >>>>>>>>>>>> On 21-Jun-2019, at 4:37 AM, Ashish Sharma < >>>>>>>> sharma.ashish960(a)gmail.com&gt; >>>>>>>>>>> wrote: >>>>>>>>>>>> >>>>>>>>>>>> Hi everyone, >>>>>>>>>>>> >>>>>>>>>>>> Update on HELM project:- >>>>>>>>>>>> >>>>>>>>>>>> We have added some features to the chart that includes:- >>>>>>>>>>>> * Making chart configurable with both mysql and postgres > database >>>>>>>>>>>> * We are now using configmaps and secrets for storing

>>>>>>>>>>> credentials >>>>>>>>>>>> * We use helm dependency to manage the dependencies of the > chart >>>>>>>>>>>> >>>>>>>>>>>> I think our chart is ready to be deployed in the incubator >>> section >>>>> of >>>>>>>>>>> helm >>>>>>>>>>>> community repo. <https://github.com/helm/charts> If you guys >>>>> agree I >>>>>>>>>>> would >>>>>>>>>>>> make the necessary changes required to deploy it there. >>>>>>>>>>>> >>>>>>>>>>>> Thanks >>>>>>>>>>>> - Ashish >>>>>>>>>>>> >>>>>>>>>>>> On Wed, Jun 5, 2019 at 7:40 PM Vincent Massol < >>> vincent(a)massol.net&gt; >>>>>>>>>>> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Hi Ashish, >>>>>>>>>>>>> >>>>>>>>>>>>>> On 5 Jun 2019, at 15:07, Ashish Sharma < >>>>> sharma.ashish960(a)gmail.com >>>>>>>>> >>>>>>>>>>>>> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>> Hi everyone, >>>>>>>>>>>>>> >>>>>>>>>>>>>> The first week update of helm chart xwiki project. We > currently >>>>>>>> have a >>>>>>>>>>>>>> chart that would deploy xwiki mysql stateful service with a >>>>>>>> persistent >>>>>>>>>>>>>> volume. The link to the repo:- >>>>>>>>>>>>>> >>>>>>>>>>>>>> https://github.com/xwiki-contrib/xwiki-helm >>>>>>>>>>>>> >>>>>>>>>>>>> Thanks for the update. >>>>>>>>>>>>> >>>>>>>>>>>>> Would be great if you could update the README file to

>>> who >>>>> to >>>>>>>>>>> use >>>>>>>>>>>>> it. >>>>>>>>>>>>> >>>>>>>>>>>>> Thanks >>>>>>>>>>>>> -Vincent >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Thank You >>>>>>>>>>>>>> Ashish >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Thu, 23 May 2019 at 7:27 PM, Ashish Sharma < >>>>>>>>>>>>> sharma.ashish960(a)gmail.com&gt; >>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Wed, May 22, 2019 at 12:42 PM Vincent Massol < >>>>>>>> vincent(a)massol.net&gt; >>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Hi Ashish, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On 21 May 2019, at 19:37, Ashish Sharma < >>>>>>>>>>> sharma.ashish960(a)gmail.com&gt; >>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Hi everyone, >>>>>>>>>>>>>>>>> Thanks for having me here >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> About Me >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> I am Ashish Sharma, selected as a student for Google > Summer >>> of >>>>>>>>>>> Code. I >>>>>>>>>>>>>>>> am >>>>>>>>>>>>>>>>> final year student enrolled in Guru Gobind Singh >>> Indraprastha >>>>>>>>>>>>>>>> University, >>>>>>>>>>>>>>>>> Delhi. I am a resident of India. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Profiles >>>>>>>>>>>>>>>>> GitHub -

>>>>>>>>>>>>>>>>> LinkedIn - https://www.linkedin.com/in/ashish932/ >>>>>>>>>>>>>>>>> Riot - @ashish932:matrix.org >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> I will be presenting my project "Helm Chart for XWiki"

>>> all >>>>> of >>>>>>>>>>> you. >>>>>>>>>>>>>>>>> Following >>>>>>>>>>>>>>>>> are the relevant details. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Helm Chart for XWiki >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Mentors: Shubham Jain, Neha Gupta >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Technologies: Kubernetes, Docker, other if required >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Overview >>>>>>>>>>>>>>>>> The proposed project is a helm chart that would deploy > xwiki >>>>> as >>>>>>>>>>> highly >>>>>>>>>>>>>>>>> available and reliable. It should be configurable with >>>>> different >>>>>>>>>>>>>>>>> databases(either a standalone database or a clustered

>>>>> that >>>>>>>> are >>>>>>>>>>>>>>>>> configurable with xwiki. It would give the option to > either >>>>>>>>>>> configure >>>>>>>>>>>>>>>> solr >>>>>>>>>>>>>>>>> externally (standalone or clustered) or managed within

>>>>>>>>>>> container. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> It also needs to be deployed on various Servlet

>>> and >>>>>>>> with >>>>>>>>>>> a >>>>>>>>>>>>>>>> LibreOffice server configured. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> In short, it should support all the configurations > supported >>> by >>>>>>>> the >>>>>>>>>>>>>>>> official XWiki Docker images :) >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> It >>>>>>>>>>>>>>>>> should deploy the app on a shared file system like a

>>> It >>>>>>>>>>> should >>>>>>>>>>>>>>>>> support Istio virtual services, istio matrix, and istio >>>>>>>> distributed >>>>>>>>>>>>>>>> tracing >>>>>>>>>>>>>>>>> and should be a secured system with RBAC and security >>>>> credential >>>>>>>>>>>>>>>> rotation. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Ok I have no knowledge about this… The shared file system >>>>> sounds >>>>>>>>>>> good >>>>>>>>>>>>> to >>>>>>>>>>>>>>>> support clustering XWiki for the permanent directory > sharing. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> The chart should be easily deployed on GKE and amazon

>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On locally on minikube for testing I guess? >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Yes we can test using minikube one >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Features >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> -> Support for different Databases >>>>>>>>>>>>>>>>> -> Choice between using an external database, a single > node >>>>> DB >>>>>>>> or >>>>>>>>>>> a >>>>>>>>>>>>>>>>> multi-cluster DB setup >>>>>>>>>>>>>>>>> -> Support for shared file system >>>>>>>>>>>>>>>>> -> Support for istio and it's services >>>>>>>>>>>>>>>>> -> RBAC, SSL and other security methods >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> If you have any features in mind that should be added > please >>>>>>>> feel >>>>>>>>>>> free >>>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>>> reply to this mail. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Some Design Questions? >>>>>>>>>>>>>>>>> -> Which Databases should be supported? >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> See >>>>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>> >>>>>>>> >>>>> >>> >

>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> More generally see the XWiki Docker image documentation

>>>>>>>>>>>>>>>> >>> https://dev.xwiki.org/xwiki/bin/view/Community/SupportStrategy >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> -> As we have to detach solr out of the docker > container(run >>>>> it >>>>>>>> in >>>>>>>>>>> an >>>>>>>>>>>>>>>>> independent container) would be there a requirement for

>>> code >>>>>>>>>>> change, >>>>>>>>>>>>>>>> and >>>>>>>>>>>>>>>>> we should approach it? >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> See the doc for Solr which explains how to configure an >>>>> external >>>>>>>>>>> SOLR. >>>>>>>>>>>>>>>> There are some XWiki code/config to deploy in the

>>> SOLR >>>>>>>>>>> AFAIR. >>>>>>>>>>>>>>>> comunity controls chart >>>>>>>>>>>>>>>>> -> Apart from solr is there any other stateful service > that >>>>>>>> could >>>>>>>>>>> or >>>>>>>>>>>>>>>> should >>>>>>>>>>>>>>>>> be detached from the docker container? >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Which docker container are you referring to? The Servlet >>>>>>>> Container >>>>>>>>>>> one? >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> xwiki:lts-mysql-tomcat >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Here is my current repository which deploys XWiki for > MySQL >>>>>>>>>>> database >>>>>>>>>>>>>>>> using >>>>>>>>>>>>>>>>> official XWiki docker container:- >>>>>>>>>>>>>>>>> https://github.com/ashish932/xwiki-helm-chart/ >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Will need to have a look. Some additional questions: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> 1) Is there an official place for publishing helm charts, >>>>>>>> similar to >>>>>>>>>>>>>>>> dockerhub for docker images? How do you provide the helm >>> charts >>>>>>>> to >>>>>>>>>>>>> XWiki >>>>>>>>>>>>>>>> users in term of packaging? In other words, what will > we/you >>>>>>>>>>> document >>>>>>>>>>>>> on >>>>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>> >>>>>>>> >>>>> >>> >

>>>>>>>>>>>>>>>> ? >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Yes, There is a centralized community control chart >>> management >>>>>>>>>>>>>>> repository. Helm charts are hosted there. Organizations

>>>>>>>> mysql, >>>>>>>>>>>>>>> wordpress uses it. >>>>>>>>>>>>>>> https://github.com/helm/charts >>>>>>>>>>>>>>> But in addition to that, we can build our own charts >>> repository >>>>>>>>>>> locally, >>>>>>>>>>>>>>> and host it. Similar to docker run command, we have helm >>> update >>>>>>>>>>> command >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> 2) How do you plan to have smoke tests for the XWiki Helm >>>>>>>> charts, as >>>>>>>>>>>>> part >>>>>>>>>>>>>>>> of the build, to verify that they work fine? How can we >>> execute >>>>>>>>>>> XWik’s >>>>>>>>>>>>>>>> functional tests based on TestContainers using Helm

>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> There are some tools available >>>>>>>>>>>>>>> chart-testing(started by helm) -98 commits >>>>>>>>>>>>>>> terratest - 1039 commits (It looks a good tool to use) >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> 3) Do the helm charts for XWiki need to be built? If so, > can >>> it >>>>>>>> be >>>>>>>>>>> done >>>>>>>>>>>>>>>> with Maven? Since we’ll be offering several I guess, how

>>> you >>>>>>>>>>> avoid >>>>>>>>>>>>> code >>>>>>>>>>>>>>>> duplication and make maintenance of them as simple as >>> possible? >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> No, the charts don’t need to be built externally, though

>>> gets >>>>>>>>>>>>>>> compressed when pushed f we build to chart repository(this > is >>> an >>>>>>>>>>>>> automatic >>>>>>>>>>>>>>> step in case of official chart repo and we can automate > using >>> ci >>>>>>>> our >>>>>>>>>>> own >>>>>>>>>>>>>>> chart repo) >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> 4) Could you provide a roadmap with various milestones and >>>>> dates? >>>>>>>> See >>>>>>>>>>>>> the >>>>>>>>>>>>>>>> last roadmap email I sent on the devs list and please

>>> to >>>>>>>> it. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Okay I would work on that and update the progress on the > mail >>>>>>>> thread >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> 5) It seems you missed the bonding period so you’ll need

>>>>> catch >>>>>>>>>>> up on >>>>>>>>>>>>>>>> that. Could you pick some jira issue and work on them >>> quickly? >>>>>>>> This >>>>>>>>>>> is >>>>>>>>>>>>> to >>>>>>>>>>>>>>>> make sure you use the proper xwiki communication

>>>>> learn >>>>>>>> to >>>>>>>>>>>>> work >>>>>>>>>>>>>>>> with the xwiki team and start understanding the xwiki >>>>>>>> architecture, >>>>>>>>>>>>> and how >>>>>>>>>>>>>>>> extensions are developed for XWiki. Since you’re a bit > late, >>>>>>>> you’ll >>>>>>>>>>>>> need to >>>>>>>>>>>>>>>> catch up and achieve this very quickly now and be present >>> full >>>>>>>> time >>>>>>>>>>> on >>>>>>>>>>>>> the >>>>>>>>>>>>>>>> xwiki chat :) >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I would start doing that >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> 6) Idea: please review the issues on the XWiki Docker >>> project’s >>>>>>>>>>> jira. >>>>>>>>>>>>>>>> Maybe you can help with those too as part of >>>>> onboarding/bonding. >>>>>>>> It >>>>>>>>>>>>> should >>>>>>>>>>>>>>>> also give you additional use cases for the XWiki Helm > Charts >>>>> IMO. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Thanks! I’m eager to follow your work and see xwiki helm >>> charts >>>>>>>> in >>>>>>>>>>>>> action. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> -Vincent >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Thank You >>>>>>>>>>>>>>>>> Ashish Sharma >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Thank You >>>>>>>>>>>>>>> Ashish >>>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>> >>>>>>>> >>>>> >>> >