func (delegate *delegate) saveInput(logger lager.Logger, status exec.ExitStatus, plan atc.GetPlan, info *exec.VersionInfo, origin event.Origin) { var version atc.Version var metadata []atc.MetadataField if info != nil && plan.Pipeline != "" { savedVR, err := delegate.db.SaveBuildInput(delegate.buildID, db.BuildInput{ Name: plan.Name, VersionedResource: vrFromInput(plan, *info), }) if err != nil { logger.Error("failed-to-save-input", err) } version = atc.Version(savedVR.Version) metadata = dbMetadataToATCMetadata(savedVR.Metadata) } ev := event.FinishGet{ Origin: origin, Plan: event.GetPlan{ Name: plan.Name, Resource: plan.Resource, Type: plan.Type, Version: plan.Version, }, ExitStatus: int(status), FetchedVersion: version, FetchedMetadata: metadata, } err := delegate.db.SaveBuildEvent(delegate.buildID, ev) if err != nil { logger.Error("failed-to-save-input-event", err) } }
func (scanner *resourceScanner) Scan(logger lager.Logger, resourceName string) error { vr, _, err := scanner.db.GetLatestVersionedResource(resourceName) if err != nil { logger.Error("failed-to-get-current-version", err) return err } return swallowErrResourceScriptFailed( scanner.ScanFromVersion(logger, resourceName, atc.Version(vr.Version)), ) }
func BuildInput(input db.BuildInput, config config.JobInput, source atc.Source) atc.BuildInput { return atc.BuildInput{ Name: input.Name, Resource: input.Resource, Type: input.Type, Source: source, Params: config.Params, Version: atc.Version(input.Version), Tags: config.Tags, } }
func PublicBuildInput(input db.BuildInput) atc.PublicBuildInput { metadata := make([]atc.MetadataField, 0, len(input.Metadata)) for _, meta := range input.Metadata { metadata = append(metadata, atc.MetadataField{ Name: meta.Name, Value: meta.Value, }) } return atc.PublicBuildInput{ Name: input.Name, Resource: input.Resource, Type: input.Type, Version: atc.Version(input.Version), Metadata: metadata, PipelineName: input.PipelineName, FirstOccurrence: input.FirstOccurrence, } }
func SavedVersionedResource(svr db.SavedVersionedResource) atc.VersionedResource { var metadata []atc.MetadataField for _, v := range svr.Metadata { metadata = append(metadata, atc.MetadataField{ Name: v.Name, Value: v.Value, }) } return atc.VersionedResource{ ID: svr.ID, PipelineName: svr.PipelineName, Resource: svr.Resource, Type: svr.Type, Version: atc.Version(svr.Version), Metadata: metadata, } }
func (scanner *resourceScanner) Run(logger lager.Logger, resourceName string) (time.Duration, error) { resourceConfig, resourceTypes, err := scanner.getResourceConfig(logger, resourceName) if err != nil { return 0, err } savedResource, found, err := scanner.db.GetResource(resourceConfig.Name) if err != nil { return 0, err } if !found { return 0, db.ResourceNotFoundError{Name: resourceConfig.Name} } interval, err := scanner.checkInterval(resourceConfig) if err != nil { setErr := scanner.db.SetResourceCheckError(savedResource, err) if setErr != nil { logger.Error("failed-to-set-check-error", err) } return 0, err } leaseLogger := logger.Session("lease", lager.Data{ "resource": resourceName, }) lease, leased, err := scanner.db.LeaseResourceChecking(logger, resourceName, interval, false) if err != nil { leaseLogger.Error("failed-to-get-lease", err, lager.Data{ "resource": resourceName, }) return interval, ErrFailedToAcquireLease } if !leased { leaseLogger.Debug("did-not-get-lease") return interval, ErrFailedToAcquireLease } vr, _, err := scanner.db.GetLatestVersionedResource(resourceName) if err != nil { logger.Error("failed-to-get-current-version", err) return interval, err } err = swallowErrResourceScriptFailed( scanner.scan(logger.Session("tick"), resourceConfig, resourceTypes, savedResource, atc.Version(vr.Version)), ) lease.Break() if err != nil { return interval, err } return interval, nil }
func (scanner *resourceScanner) scan( logger lager.Logger, resourceConfig atc.ResourceConfig, resourceTypes atc.ResourceTypes, savedResource db.SavedResource, fromVersion atc.Version, ) error { pipelinePaused, err := scanner.db.IsPaused() if err != nil { logger.Error("failed-to-check-if-pipeline-paused", err) return err } if pipelinePaused { logger.Debug("pipeline-paused") return nil } if savedResource.Paused { logger.Debug("resource-paused") return nil } pipelineID := scanner.db.GetPipelineID() var resourceTypeVersion atc.Version _, found := resourceTypes.Lookup(resourceConfig.Type) if found { savedResourceType, resourceTypeFound, err := scanner.db.GetResourceType(resourceConfig.Type) if err != nil { logger.Error("failed-to-find-resource-type", err) return err } if resourceTypeFound { resourceTypeVersion = atc.Version(savedResourceType.Version) } } session := resource.Session{ ID: worker.Identifier{ ResourceTypeVersion: resourceTypeVersion, ResourceID: savedResource.ID, Stage: db.ContainerStageRun, CheckType: resourceConfig.Type, CheckSource: resourceConfig.Source, }, Metadata: worker.Metadata{ Type: db.ContainerTypeCheck, PipelineID: pipelineID, }, Ephemeral: true, } res, err := scanner.tracker.Init( logger, resource.TrackerMetadata{ ResourceName: resourceConfig.Name, PipelineName: savedResource.PipelineName, ExternalURL: scanner.externalURL, }, session, resource.ResourceType(resourceConfig.Type), []string{}, resourceTypes, worker.NoopImageFetchingDelegate{}, ) if err != nil { logger.Error("failed-to-initialize-new-resource", err) return err } defer res.Release(nil) logger.Debug("checking", lager.Data{ "from": fromVersion, }) newVersions, err := res.Check(resourceConfig.Source, fromVersion) setErr := scanner.db.SetResourceCheckError(savedResource, err) if setErr != nil { logger.Error("failed-to-set-check-error", err) } if err != nil { if rErr, ok := err.(resource.ErrResourceScriptFailed); ok { logger.Info("check-failed", lager.Data{"exit-status": rErr.ExitStatus}) return rErr } logger.Error("failed-to-check", err) return err } if len(newVersions) == 0 { logger.Debug("no-new-versions") return nil } logger.Info("versions-found", lager.Data{ "versions": newVersions, "total": len(newVersions), }) err = scanner.db.SaveResourceVersions(resourceConfig, newVersions) if err != nil { logger.Error("failed-to-save-versions", err, lager.Data{ "versions": newVersions, }) } return nil }
func (factory *BuildFactory) constructPlanFromConfig( planConfig atc.PlanConfig, resources atc.ResourceConfigs, inputs []db.BuildInput, hasHooks bool, ) atc.Plan { var plan atc.Plan switch { case planConfig.Do != nil: if hasHooks { plan = factory.constructPlanHookBasedPlan( *planConfig.Do, resources, inputs, ) } else { plan = factory.constructPlanSequenceBasedPlan( *planConfig.Do, resources, inputs, ) } case planConfig.Put != "": logicalName := planConfig.Put resourceName := planConfig.Resource if resourceName == "" { resourceName = logicalName } resource, _ := resources.Lookup(resourceName) putPlan := &atc.PutPlan{ Type: resource.Type, Name: logicalName, Pipeline: factory.PipelineName, Resource: resourceName, Source: resource.Source, Params: planConfig.Params, GetParams: planConfig.GetParams, Tags: planConfig.Tags, } plan = atc.Plan{ PutGet: &atc.PutGetPlan{ Head: atc.Plan{ Put: putPlan, }, Rest: atc.Plan{}, }, } case planConfig.Get != "": resourceName := planConfig.Resource if resourceName == "" { resourceName = planConfig.Get } resource, _ := resources.Lookup(resourceName) name := planConfig.Get var version db.Version for _, input := range inputs { if input.Name == name { version = input.Version break } } plan = atc.Plan{ Get: &atc.GetPlan{ Type: resource.Type, Name: name, Pipeline: factory.PipelineName, Resource: resourceName, Source: resource.Source, Params: planConfig.Params, Version: atc.Version(version), Tags: planConfig.Tags, }, } case planConfig.Task != "": plan = atc.Plan{ Task: &atc.TaskPlan{ Name: planConfig.Task, Privileged: planConfig.Privileged, Config: planConfig.TaskConfig, ConfigPath: planConfig.TaskConfigPath, Tags: planConfig.Tags, }, } case planConfig.Try != nil: plan = atc.Plan{ Try: &atc.TryPlan{ Step: factory.constructPlanFromConfig( *planConfig.Try, resources, inputs, hasHooks, ), }, } case planConfig.Aggregate != nil: aggregate := atc.AggregatePlan{} for _, planConfig := range *planConfig.Aggregate { aggregate = append(aggregate, factory.constructPlanFromConfig( planConfig, resources, inputs, hasHooks, )) } plan = atc.Plan{ Aggregate: &aggregate, } } if planConfig.Conditions != nil { plan = atc.Plan{ Conditional: &atc.ConditionalPlan{ Conditions: *planConfig.Conditions, Plan: plan, }, } } if planConfig.Timeout != 0 { plan = atc.Plan{ Timeout: &atc.TimeoutPlan{ Duration: planConfig.Timeout, Step: plan, }, } } hooks := false failurePlan := atc.Plan{} if planConfig.Failure != nil { hooks = true failurePlan = factory.constructPlanFromConfig(*planConfig.Failure, resources, inputs, hasHooks) } ensurePlan := atc.Plan{} if planConfig.Ensure != nil { hooks = true ensurePlan = factory.constructPlanFromConfig(*planConfig.Ensure, resources, inputs, hasHooks) } successPlan := atc.Plan{} if planConfig.Success != nil { hooks = true successPlan = factory.constructPlanFromConfig(*planConfig.Success, resources, inputs, hasHooks) } if hooks { plan = atc.Plan{ HookedCompose: &atc.HookedComposePlan{ Step: plan, OnFailure: failurePlan, OnCompletion: ensurePlan, OnSuccess: successPlan, }, } } return plan }
func (factory *BuildFactory) constructPlanFromConfig( planConfig atc.PlanConfig, resources atc.ResourceConfigs, inputs []db.BuildInput, hasHooks bool, ) atc.Plan { var plan atc.Plan switch { case planConfig.Do != nil: if hasHooks { plan = factory.constructPlanHookBasedPlan( *planConfig.Do, resources, inputs, ) } else { plan = factory.constructPlanSequenceBasedPlan( *planConfig.Do, resources, inputs, ) } if plan.Location == nil { plan.Location = planConfig.Location } case planConfig.Put != "": logicalName := planConfig.Put resourceName := planConfig.Resource if resourceName == "" { resourceName = logicalName } resource, _ := resources.Lookup(resourceName) putPlan := &atc.PutPlan{ Type: resource.Type, Name: logicalName, Pipeline: factory.PipelineName, Resource: resourceName, Source: resource.Source, Params: planConfig.Params, Tags: planConfig.Tags, } dependentGetPlan := &atc.DependentGetPlan{ Type: resource.Type, Name: logicalName, Pipeline: factory.PipelineName, Resource: resourceName, Params: planConfig.GetParams, Tags: planConfig.Tags, Source: resource.Source, } stepLocation := &atc.Location{} nextLocation := &atc.Location{} if planConfig.Location != nil { stepLocation.ID = planConfig.Location.ID stepLocation.Hook = planConfig.Location.Hook if planConfig.Location.ParallelGroup != 0 { stepLocation.ParallelGroup = planConfig.Location.ParallelGroup } else { stepLocation.ParentID = planConfig.Location.ParentID } nextLocation.ID = stepLocation.ID + 1 nextLocation.ParentID = stepLocation.ID } plan = atc.Plan{ // Location: planConfig.Location, OnSuccess: &atc.OnSuccessPlan{ Step: atc.Plan{ Location: stepLocation, Put: putPlan, }, Next: atc.Plan{ Location: nextLocation, DependentGet: dependentGetPlan, }, }, } case planConfig.Get != "": resourceName := planConfig.Resource if resourceName == "" { resourceName = planConfig.Get } resource, _ := resources.Lookup(resourceName) name := planConfig.Get var version db.Version for _, input := range inputs { if input.Name == name { version = input.Version break } } plan = atc.Plan{ Location: planConfig.Location, Get: &atc.GetPlan{ Type: resource.Type, Name: name, Pipeline: factory.PipelineName, Resource: resourceName, Source: resource.Source, Params: planConfig.Params, Version: atc.Version(version), Tags: planConfig.Tags, }, } case planConfig.Task != "": plan = atc.Plan{ Location: planConfig.Location, Task: &atc.TaskPlan{ Name: planConfig.Task, Privileged: planConfig.Privileged, Config: planConfig.TaskConfig, ConfigPath: planConfig.TaskConfigPath, Tags: planConfig.Tags, }, } case planConfig.Try != nil: nextStep := factory.constructPlanFromConfig( *planConfig.Try, resources, inputs, hasHooks) plan = atc.Plan{ Location: planConfig.Location, Try: &atc.TryPlan{ Step: nextStep, }, } case planConfig.Aggregate != nil: aggregate := atc.AggregatePlan{} for _, planConfig := range *planConfig.Aggregate { nextStep := factory.constructPlanFromConfig( planConfig, resources, inputs, hasHooks) aggregate = append(aggregate, nextStep) } plan = atc.Plan{ Location: planConfig.Location, Aggregate: &aggregate, } } if planConfig.Conditions != nil { plan = atc.Plan{ Conditional: &atc.ConditionalPlan{ Conditions: *planConfig.Conditions, Plan: plan, }, } } if planConfig.Timeout != "" { plan = atc.Plan{ Timeout: &atc.TimeoutPlan{ Duration: planConfig.Timeout, Step: plan, }, } } constructionParams := factory.ensureIfPresent(factory.successIfPresent(factory.failureIfPresent( constructionParams{ plan: plan, planConfig: planConfig, resources: resources, inputs: inputs, hasHooks: hasHooks, })), ) return constructionParams.plan }
func (factory *buildFactory) constructUnhookedPlan( planConfig atc.PlanConfig, resources atc.ResourceConfigs, resourceTypes atc.ResourceTypes, inputs []db.BuildInput, ) (atc.Plan, error) { var plan atc.Plan var err error switch { case planConfig.Do != nil: plan, err = factory.do( *planConfig.Do, resources, resourceTypes, inputs, ) if err != nil { return atc.Plan{}, err } case planConfig.Put != "": logicalName := planConfig.Put resourceName := planConfig.Resource if resourceName == "" { resourceName = logicalName } resource, found := resources.Lookup(resourceName) if !found { return atc.Plan{}, ErrResourceNotFound } putPlan := atc.PutPlan{ Type: resource.Type, Name: logicalName, PipelineID: factory.PipelineID, Resource: resourceName, Source: resource.Source, Params: planConfig.Params, Tags: planConfig.Tags, ResourceTypes: resourceTypes, } dependentGetPlan := atc.DependentGetPlan{ Type: resource.Type, Name: logicalName, PipelineID: factory.PipelineID, Resource: resourceName, Params: planConfig.GetParams, Tags: planConfig.Tags, Source: resource.Source, ResourceTypes: resourceTypes, } plan = factory.planFactory.NewPlan(atc.OnSuccessPlan{ Step: factory.planFactory.NewPlan(putPlan), Next: factory.planFactory.NewPlan(dependentGetPlan), }) case planConfig.Get != "": resourceName := planConfig.Resource if resourceName == "" { resourceName = planConfig.Get } resource, found := resources.Lookup(resourceName) if !found { return atc.Plan{}, ErrResourceNotFound } name := planConfig.Get var version db.Version for _, input := range inputs { if input.Name == name { version = input.Version break } } plan = factory.planFactory.NewPlan(atc.GetPlan{ Type: resource.Type, Name: name, PipelineID: factory.PipelineID, Resource: resourceName, Source: resource.Source, Params: planConfig.Params, Version: atc.Version(version), Tags: planConfig.Tags, ResourceTypes: resourceTypes, }) case planConfig.Task != "": plan = factory.planFactory.NewPlan(atc.TaskPlan{ Name: planConfig.Task, PipelineID: factory.PipelineID, Privileged: planConfig.Privileged, Config: planConfig.TaskConfig, ConfigPath: planConfig.TaskConfigPath, Tags: planConfig.Tags, ResourceTypes: resourceTypes, Params: planConfig.Params, InputMapping: planConfig.InputMapping, OutputMapping: planConfig.OutputMapping, }) case planConfig.Try != nil: nextStep, err := factory.constructPlanFromConfig( *planConfig.Try, resources, resourceTypes, inputs, ) if err != nil { return atc.Plan{}, err } plan = factory.planFactory.NewPlan(atc.TryPlan{ Step: nextStep, }) case planConfig.Aggregate != nil: aggregate := atc.AggregatePlan{} for _, planConfig := range *planConfig.Aggregate { nextStep, err := factory.constructPlanFromConfig( planConfig, resources, resourceTypes, inputs, ) if err != nil { return atc.Plan{}, err } aggregate = append(aggregate, nextStep) } plan = factory.planFactory.NewPlan(aggregate) } if planConfig.Timeout != "" { plan = factory.planFactory.NewPlan(atc.TimeoutPlan{ Duration: planConfig.Timeout, Step: plan, }) } return plan, nil }
func (radar *Radar) scan(logger lager.Logger, resourceConfig atc.ResourceConfig, resourceTypes atc.ResourceTypes, savedResource db.SavedResource) error { pipelinePaused, err := radar.db.IsPaused() if err != nil { logger.Error("failed-to-check-if-pipeline-paused", err) return err } if pipelinePaused { logger.Debug("pipeline-paused") return nil } if savedResource.Paused { logger.Debug("resource-paused") return nil } session := resource.Session{ ID: worker.Identifier{ ResourceID: savedResource.ID, Stage: db.ContainerStageRun, CheckType: resourceConfig.Type, CheckSource: resourceConfig.Source, }, Metadata: worker.Metadata{ Type: db.ContainerTypeCheck, PipelineName: radar.db.GetPipelineName(), }, Ephemeral: true, } res, err := radar.tracker.Init( logger, resource.EmptyMetadata{}, session, resource.ResourceType(resourceConfig.Type), []string{}, resourceTypes, worker.NoopImageFetchingDelegate{}, ) if err != nil { logger.Error("failed-to-initialize-new-resource", err) return err } defer res.Release(nil) vr, found, err := radar.db.GetLatestVersionedResource(savedResource) if err != nil { logger.Error("failed-to-get-current-version", err) return err } var from db.Version if found { from = vr.Version } logger.Debug("checking", lager.Data{ "from": from, }) newVersions, err := res.Check(resourceConfig.Source, atc.Version(from)) setErr := radar.db.SetResourceCheckError(savedResource, err) if setErr != nil { logger.Error("failed-to-set-check-error", err) } if err != nil { if rErr, ok := err.(resource.ErrResourceScriptFailed); ok { logger.Info("check-failed", lager.Data{"exit-status": rErr.ExitStatus}) return nil } logger.Error("failed-to-check", err) return err } if len(newVersions) == 0 { logger.Debug("no-new-versions") return nil } logger.Info("versions-found", lager.Data{ "versions": newVersions, "total": len(newVersions), }) err = radar.db.SaveResourceVersions(resourceConfig, newVersions) if err != nil { logger.Error("failed-to-save-versions", err, lager.Data{ "versions": newVersions, }) } return nil }
func VersionedResource(vr db.VersionedResource) atc.VersionedResource { return atc.VersionedResource{ Resource: vr.Resource, Version: atc.Version(vr.Version), } }
func (radar *Radar) scan(logger lager.Logger, resourceConfig atc.ResourceConfig, savedResource db.SavedResource) error { pipelinePaused, err := radar.db.IsPaused() if err != nil { logger.Error("failed-to-check-if-pipeline-paused", err) return err } if pipelinePaused { logger.Debug("pipeline-paused") return nil } if savedResource.Paused { logger.Debug("resource-paused") return nil } typ := resource.ResourceType(resourceConfig.Type) res, err := radar.tracker.Init( logger, resource.EmptyMetadata{}, checkIdentifier(radar.db.GetPipelineName(), resourceConfig), typ, []string{}, ) if err != nil { logger.Error("failed-to-initialize-new-resource", err) return err } defer res.Release(0) vr, found, err := radar.db.GetLatestVersionedResource(savedResource) if err != nil { logger.Error("failed-to-get-current-version", err) return err } var from db.Version if found { from = vr.Version } logger.Debug("checking", lager.Data{ "from": from, }) newVersions, err := res.Check(resourceConfig.Source, atc.Version(from)) setErr := radar.db.SetResourceCheckError(savedResource, err) if setErr != nil { logger.Error("failed-to-set-check-error", err) } if err != nil { if rErr, ok := err.(resource.ErrResourceScriptFailed); ok { logger.Info("check-failed", lager.Data{"exit-status": rErr.ExitStatus}) return nil } logger.Error("failed-to-check", err) return err } if len(newVersions) == 0 { logger.Debug("no-new-versions") return nil } logger.Info("versions-found", lager.Data{ "versions": newVersions, "total": len(newVersions), }) err = radar.db.SaveResourceVersions(resourceConfig, newVersions) if err != nil { logger.Error("failed-to-save-versions", err, lager.Data{ "versions": newVersions, }) } return nil }
func (radar *Radar) scan(logger lager.Logger, resourceName string) error { pipelinePaused, err := radar.db.IsPaused() if err != nil { logger.Error("failed-to-check-if-pipeline-paused", err) return err } if pipelinePaused { logger.Debug("pipeline-paused") return nil } config, _, err := radar.db.GetConfig() if err != nil { logger.Error("failed-to-get-config", err) // don't propagate error; we can just retry next tick return nil } resourceConfig, found := config.Resources.Lookup(resourceName) if !found { logger.Info("resource-removed-from-configuration") // return an error so that we exit return resourceNotConfiguredError{ResourceName: resourceName} } savedResource, err := radar.db.GetResource(resourceName) if err != nil { return err } if savedResource.Paused { return nil } typ := resource.ResourceType(resourceConfig.Type) res, err := radar.tracker.Init(checkIdentifier(radar.db.GetPipelineName(), resourceConfig), typ, []string{}) if err != nil { logger.Error("failed-to-initialize-new-resource", err) return err } defer res.Release() var from db.Version if vr, err := radar.db.GetLatestVersionedResource(savedResource); err == nil { from = vr.Version } logger.Debug("checking", lager.Data{ "from": from, }) newVersions, err := res.Check(resourceConfig.Source, atc.Version(from)) setErr := radar.db.SetResourceCheckError(savedResource, err) if setErr != nil { logger.Error("failed-to-set-check-error", err) } if err != nil { logger.Error("failed-to-check", err) return err } if len(newVersions) == 0 { logger.Debug("no-new-versions") return nil } logger.Info("versions-found", lager.Data{ "versions": newVersions, "total": len(newVersions), }) err = radar.db.SaveResourceVersions(resourceConfig, newVersions) if err != nil { logger.Error("failed-to-save-versions", err, lager.Data{ "versions": newVersions, }) } return nil }