func hasMorphism(via interface{}, nodes ...string) morphism { return morphism{ Name: "has", Reversal: func() morphism { return hasMorphism(via, nodes...) }, Apply: func(qs graph.QuadStore, it graph.Iterator) graph.Iterator { var sub graph.Iterator if len(nodes) == 0 { sub = qs.NodesAllIterator() } else { fixed := qs.FixedIterator() for _, n := range nodes { fixed.Add(qs.ValueOf(n)) } sub = fixed } var viaPath *Path if via != nil { viaPath = buildViaPath(qs, via) } else { viaPath = buildViaPath(qs) } subAnd := iterator.NewAnd(qs) subAnd.AddSubIterator(iterator.NewLinksTo(qs, sub, quad.Object)) subAnd.AddSubIterator(iterator.NewLinksTo(qs, viaPath.BuildIterator(), quad.Predicate)) hasa := iterator.NewHasA(qs, subAnd, quad.Subject) and := iterator.NewAnd(qs) and.AddSubIterator(it) and.AddSubIterator(hasa) return and }, } }
func TestIteratorsAndNextResultOrderA(t *testing.T) { glog.Info("\n-----------\n") inst, opts, err := createInstance() defer inst.Close() if err != nil { t.Fatalf("failed to create instance: %v", err) } qs, _, _ := makeTestStore(simpleGraph, opts) if qs.Size() != 11 { t.Fatal("Incorrect number of quads") } fixed := qs.FixedIterator() fixed.Add(qs.ValueOf("C")) fixed2 := qs.FixedIterator() fixed2.Add(qs.ValueOf("follows")) all := qs.NodesAllIterator() innerAnd := iterator.NewAnd(qs) innerAnd.AddSubIterator(iterator.NewLinksTo(qs, fixed2, quad.Predicate)) innerAnd.AddSubIterator(iterator.NewLinksTo(qs, all, quad.Object)) hasa := iterator.NewHasA(qs, innerAnd, quad.Subject) outerAnd := iterator.NewAnd(qs) outerAnd.AddSubIterator(fixed) outerAnd.AddSubIterator(hasa) if !outerAnd.Next() { t.Error("Expected one matching subtree") } val := outerAnd.Result() if qs.NameOf(val) != "C" { t.Errorf("Matching subtree should be %s, got %s", "barak", qs.NameOf(val)) } var ( got []string expect = []string{"B", "D"} ) for { got = append(got, qs.NameOf(all.Result())) if !outerAnd.NextPath() { break } } sort.Strings(got) if !reflect.DeepEqual(got, expect) { t.Errorf("Unexpected result, got:%q expect:%q", got, expect) } if outerAnd.Next() { t.Error("More than one possible top level output?") } }
func TestIteratorsAndNextResultOrderA(t *testing.T) { qs, _, _ := makeTestStore(simpleGraph) fixed := qs.FixedIterator() fixed.Add(qs.ValueOf("C")) fixed2 := qs.FixedIterator() fixed2.Add(qs.ValueOf("follows")) all := qs.NodesAllIterator() innerAnd := iterator.NewAnd(qs) innerAnd.AddSubIterator(iterator.NewLinksTo(qs, fixed2, quad.Predicate)) innerAnd.AddSubIterator(iterator.NewLinksTo(qs, all, quad.Object)) hasa := iterator.NewHasA(qs, innerAnd, quad.Subject) outerAnd := iterator.NewAnd(qs) outerAnd.AddSubIterator(fixed) outerAnd.AddSubIterator(hasa) if !outerAnd.Next() { t.Error("Expected one matching subtree") } val := outerAnd.Result() if qs.NameOf(val) != "C" { t.Errorf("Matching subtree should be %s, got %s", "barak", qs.NameOf(val)) } var ( got []string expect = []string{"B", "D"} ) for { got = append(got, qs.NameOf(all.Result())) if !outerAnd.NextPath() { break } } sort.Strings(got) if !reflect.DeepEqual(got, expect) { t.Errorf("Unexpected result, got:%q expect:%q", got, expect) } if outerAnd.Next() { t.Error("More than one possible top level output?") } }
func TestRemoveQuad(t *testing.T) { qs, w, _ := makeTestStore(simpleGraph) w.RemoveQuad(quad.Quad{ Subject: "E", Predicate: "follows", Object: "F", Label: "", }) fixed := qs.FixedIterator() fixed.Add(qs.ValueOf("E")) fixed2 := qs.FixedIterator() fixed2.Add(qs.ValueOf("follows")) innerAnd := iterator.NewAnd(qs) innerAnd.AddSubIterator(iterator.NewLinksTo(qs, fixed, quad.Subject)) innerAnd.AddSubIterator(iterator.NewLinksTo(qs, fixed2, quad.Predicate)) hasa := iterator.NewHasA(qs, innerAnd, quad.Object) newIt, _ := hasa.Optimize() if graph.Next(newIt) { t.Error("E should not have any followers.") } }
func (qs *QuadStore) optimizeAndIterator(it *iterator.And) (graph.Iterator, bool) { // Fail fast if nothing can happen glog.V(4).Infoln("Entering optimizeAndIterator", it.UID()) found := false for _, it := range it.SubIterators() { glog.V(4).Infoln(it.Type()) if it.Type() == mongoType { found = true } } if !found { glog.V(4).Infoln("Aborting optimizeAndIterator") return it, false } newAnd := iterator.NewAnd(qs) var mongoIt *Iterator for _, it := range it.SubIterators() { switch it.Type() { case mongoType: if mongoIt == nil { mongoIt = it.(*Iterator) } else { newAnd.AddSubIterator(it) } case graph.LinksTo: continue default: newAnd.AddSubIterator(it) } } stats := mongoIt.Stats() lset := []graph.Linkage{ { Dir: mongoIt.dir, Value: qs.ValueOf(mongoIt.name), }, } n := 0 for _, it := range it.SubIterators() { if it.Type() == graph.LinksTo { lto := it.(*iterator.LinksTo) // Is it more effective to do the replacement, or let the mongo check the linksto? ltostats := lto.Stats() if (ltostats.ContainsCost+stats.NextCost)*stats.Size > (ltostats.NextCost+stats.ContainsCost)*ltostats.Size { continue } newLto := NewLinksTo(qs, lto.SubIterators()[0], "quads", lto.Direction(), lset) newAnd.AddSubIterator(newLto) n++ } } if n == 0 { return it, false } return newAnd.Optimize() }
func inOutIterator(viaPath *Path, it graph.Iterator, reverse bool) graph.Iterator { in, out := quad.Subject, quad.Object if reverse { in, out = out, in } lto := iterator.NewLinksTo(viaPath.qs, it, in) and := iterator.NewAnd(viaPath.qs) and.AddSubIterator(iterator.NewLinksTo(viaPath.qs, viaPath.BuildIterator(), quad.Predicate)) and.AddSubIterator(lto) return iterator.NewHasA(viaPath.qs, and, out) }
func iteratorMorphism(it graph.Iterator) morphism { return morphism{ Name: "iterator", Reversal: func() morphism { return iteratorMorphism(it) }, Apply: func(qs graph.QuadStore, subIt graph.Iterator) graph.Iterator { and := iterator.NewAnd(qs) and.AddSubIterator(it) and.AddSubIterator(subIt) return and }, } }
func andMorphism(p *Path) morphism { return morphism{ Name: "and", Reversal: func() morphism { return andMorphism(p) }, Apply: func(qs graph.QuadStore, it graph.Iterator) graph.Iterator { subIt := p.BuildIteratorOn(qs) and := iterator.NewAnd(qs) and.AddSubIterator(it) and.AddSubIterator(subIt) return and }, } }
func buildSave(qs graph.QuadStore, via interface{}, tag string, it graph.Iterator, reverse bool) graph.Iterator { all := qs.NodesAllIterator() all.Tagger().Add(tag) node, allDir := quad.Subject, quad.Object var viaPath *Path if via != nil { viaPath = buildViaPath(qs, via) } else { viaPath = buildViaPath(qs) } if reverse { node, allDir = allDir, node } lto := iterator.NewLinksTo(qs, all, allDir) subAnd := iterator.NewAnd(qs) subAnd.AddSubIterator(iterator.NewLinksTo(qs, viaPath.BuildIterator(), quad.Predicate)) subAnd.AddSubIterator(lto) hasa := iterator.NewHasA(qs, subAnd, node) and := iterator.NewAnd(qs) and.AddSubIterator(hasa) and.AddSubIterator(it) return and }
func exceptMorphism(p *Path) morphism { return morphism{ Name: "except", Reversal: func() morphism { return exceptMorphism(p) }, Apply: func(qs graph.QuadStore, base graph.Iterator) graph.Iterator { subIt := p.BuildIteratorOn(qs) notIt := iterator.NewNot(subIt, qs.NodesAllIterator()) and := iterator.NewAnd(qs) and.AddSubIterator(base) and.AddSubIterator(notIt) return and }, } }
func buildInOutIterator(obj *otto.Object, qs graph.QuadStore, base graph.Iterator, isReverse bool) graph.Iterator { argList, _ := obj.Get("_gremlin_values") if argList.Class() != "GoArray" { glog.Errorln("How is arglist not an array? Return nothing.", argList.Class()) return iterator.NewNull() } argArray := argList.Object() lengthVal, _ := argArray.Get("length") length, _ := lengthVal.ToInteger() var predicateNodeIterator graph.Iterator if length == 0 { predicateNodeIterator = qs.NodesAllIterator() } else { zero, _ := argArray.Get("0") predicateNodeIterator = buildIteratorFromValue(zero, qs) } if length >= 2 { var tags []string one, _ := argArray.Get("1") if one.IsString() { tags = append(tags, one.String()) } else if one.Class() == "Array" { tags = stringsFrom(one.Object()) } for _, tag := range tags { predicateNodeIterator.Tagger().Add(tag) } } in, out := quad.Subject, quad.Object if isReverse { in, out = out, in } lto := iterator.NewLinksTo(qs, base, in) and := iterator.NewAnd(qs) and.AddSubIterator(iterator.NewLinksTo(qs, predicateNodeIterator, quad.Predicate)) and.AddSubIterator(lto) return iterator.NewHasA(qs, and, out) }
func isMorphism(nodes ...string) morphism { return morphism{ Name: "is", Reversal: func() morphism { return isMorphism(nodes...) }, Apply: func(qs graph.QuadStore, it graph.Iterator) graph.Iterator { var sub graph.Iterator if len(nodes) == 0 { sub = qs.NodesAllIterator() } else { fixed := qs.FixedIterator() for _, n := range nodes { fixed.Add(qs.ValueOf(n)) } sub = fixed } and := iterator.NewAnd(qs) and.AddSubIterator(sub) and.AddSubIterator(it) return and }, } }
func TestSetIterator(t *testing.T) { tmpDir, _ := ioutil.TempDir(os.TempDir(), "cayley_test") t.Log(tmpDir) defer os.RemoveAll(tmpDir) err := createNewLevelDB(tmpDir, nil) if err != nil { t.Fatalf("Failed to create working directory") } qs, err := newQuadStore(tmpDir, nil) if qs == nil || err != nil { t.Error("Failed to create leveldb QuadStore.") } defer qs.Close() w, _ := writer.NewSingleReplication(qs, nil) w.AddQuadSet(makeQuadSet()) expect := []quad.Quad{ {"C", "follows", "B", ""}, {"C", "follows", "D", ""}, } sort.Sort(ordered(expect)) // Subject iterator. it := qs.QuadIterator(quad.Subject, qs.ValueOf("C")) if got := iteratedQuads(qs, it); !reflect.DeepEqual(got, expect) { t.Errorf("Failed to get expected results, got:%v expect:%v", got, expect) } it.Reset() and := iterator.NewAnd(qs) and.AddSubIterator(qs.QuadsAllIterator()) and.AddSubIterator(it) if got := iteratedQuads(qs, and); !reflect.DeepEqual(got, expect) { t.Errorf("Failed to get confirm expected results, got:%v expect:%v", got, expect) } // Object iterator. it = qs.QuadIterator(quad.Object, qs.ValueOf("F")) expect = []quad.Quad{ {"B", "follows", "F", ""}, {"E", "follows", "F", ""}, } sort.Sort(ordered(expect)) if got := iteratedQuads(qs, it); !reflect.DeepEqual(got, expect) { t.Errorf("Failed to get expected results, got:%v expect:%v", got, expect) } and = iterator.NewAnd(qs) and.AddSubIterator(qs.QuadIterator(quad.Subject, qs.ValueOf("B"))) and.AddSubIterator(it) expect = []quad.Quad{ {"B", "follows", "F", ""}, } if got := iteratedQuads(qs, and); !reflect.DeepEqual(got, expect) { t.Errorf("Failed to get confirm expected results, got:%v expect:%v", got, expect) } // Predicate iterator. it = qs.QuadIterator(quad.Predicate, qs.ValueOf("status")) expect = []quad.Quad{ {"B", "status", "cool", "status_graph"}, {"D", "status", "cool", "status_graph"}, {"G", "status", "cool", "status_graph"}, } sort.Sort(ordered(expect)) if got := iteratedQuads(qs, it); !reflect.DeepEqual(got, expect) { t.Errorf("Failed to get expected results from predicate iterator, got:%v expect:%v", got, expect) } // Label iterator. it = qs.QuadIterator(quad.Label, qs.ValueOf("status_graph")) expect = []quad.Quad{ {"B", "status", "cool", "status_graph"}, {"D", "status", "cool", "status_graph"}, {"G", "status", "cool", "status_graph"}, } sort.Sort(ordered(expect)) if got := iteratedQuads(qs, it); !reflect.DeepEqual(got, expect) { t.Errorf("Failed to get expected results from predicate iterator, got:%v expect:%v", got, expect) } it.Reset() // Order is important and = iterator.NewAnd(qs) and.AddSubIterator(qs.QuadIterator(quad.Subject, qs.ValueOf("B"))) and.AddSubIterator(it) expect = []quad.Quad{ {"B", "status", "cool", "status_graph"}, } if got := iteratedQuads(qs, and); !reflect.DeepEqual(got, expect) { t.Errorf("Failed to get confirm expected results, got:%v expect:%v", got, expect) } it.Reset() // Order is important and = iterator.NewAnd(qs) and.AddSubIterator(it) and.AddSubIterator(qs.QuadIterator(quad.Subject, qs.ValueOf("B"))) expect = []quad.Quad{ {"B", "status", "cool", "status_graph"}, } if got := iteratedQuads(qs, and); !reflect.DeepEqual(got, expect) { t.Errorf("Failed to get confirm expected results, got:%v expect:%v", got, expect) } }
func buildIteratorTree(tree *peg.ExpressionTree, qs graph.QuadStore) graph.Iterator { switch tree.Name { case "Start": return buildIteratorTree(tree.Children[0], qs) case "NodeIdentifier": var out graph.Iterator nodeID := getIdentString(tree) if tree.Children[0].Name == "Variable" { allIt := qs.NodesAllIterator() allIt.Tagger().Add(nodeID) out = allIt } else { n := nodeID if tree.Children[0].Children[0].Name == "ColonIdentifier" { n = nodeID[1:] } fixed := qs.FixedIterator() fixed.Add(qs.ValueOf(n)) out = fixed } return out case "PredIdentifier": i := 0 if tree.Children[0].Name == "Reverse" { //Taken care of below i++ } it := buildIteratorTree(tree.Children[i], qs) lto := iterator.NewLinksTo(qs, it, quad.Predicate) return lto case "RootConstraint": constraintCount := 0 and := iterator.NewAnd(qs) for _, c := range tree.Children { switch c.Name { case "NodeIdentifier": fallthrough case "Constraint": it := buildIteratorTree(c, qs) and.AddSubIterator(it) constraintCount++ continue default: continue } } return and case "Constraint": var hasa *iterator.HasA topLevelDir := quad.Subject subItDir := quad.Object subAnd := iterator.NewAnd(qs) isOptional := false for _, c := range tree.Children { switch c.Name { case "PredIdentifier": if c.Children[0].Name == "Reverse" { topLevelDir = quad.Object subItDir = quad.Subject } it := buildIteratorTree(c, qs) subAnd.AddSubIterator(it) continue case "PredicateKeyword": switch c.Children[0].Name { case "OptionalKeyword": isOptional = true } case "NodeIdentifier": fallthrough case "RootConstraint": it := buildIteratorTree(c, qs) l := iterator.NewLinksTo(qs, it, subItDir) subAnd.AddSubIterator(l) continue default: continue } } hasa = iterator.NewHasA(qs, subAnd, topLevelDir) if isOptional { optional := iterator.NewOptional(hasa) return optional } return hasa default: return &iterator.Null{} } }
func buildIteratorTreeHelper(obj *otto.Object, qs graph.QuadStore, base graph.Iterator) graph.Iterator { // TODO: Better error handling var ( it graph.Iterator subIt graph.Iterator ) if prev, _ := obj.Get("_gremlin_prev"); !prev.IsObject() { subIt = base } else { subIt = buildIteratorTreeHelper(prev.Object(), qs, base) } stringArgs := propertiesOf(obj, "string_args") val, _ := obj.Get("_gremlin_type") switch val.String() { case "vertex": if len(stringArgs) == 0 { it = qs.NodesAllIterator() } else { fixed := qs.FixedIterator() for _, name := range stringArgs { fixed.Add(qs.ValueOf(name)) } it = fixed } case "tag": it = subIt for _, tag := range stringArgs { it.Tagger().Add(tag) } case "save": all := qs.NodesAllIterator() if len(stringArgs) > 2 || len(stringArgs) == 0 { return iterator.NewNull() } if len(stringArgs) == 2 { all.Tagger().Add(stringArgs[1]) } else { all.Tagger().Add(stringArgs[0]) } predFixed := qs.FixedIterator() predFixed.Add(qs.ValueOf(stringArgs[0])) subAnd := iterator.NewAnd(qs) subAnd.AddSubIterator(iterator.NewLinksTo(qs, predFixed, quad.Predicate)) subAnd.AddSubIterator(iterator.NewLinksTo(qs, all, quad.Object)) hasa := iterator.NewHasA(qs, subAnd, quad.Subject) and := iterator.NewAnd(qs) and.AddSubIterator(hasa) and.AddSubIterator(subIt) it = and case "saver": all := qs.NodesAllIterator() if len(stringArgs) > 2 || len(stringArgs) == 0 { return iterator.NewNull() } if len(stringArgs) == 2 { all.Tagger().Add(stringArgs[1]) } else { all.Tagger().Add(stringArgs[0]) } predFixed := qs.FixedIterator() predFixed.Add(qs.ValueOf(stringArgs[0])) subAnd := iterator.NewAnd(qs) subAnd.AddSubIterator(iterator.NewLinksTo(qs, predFixed, quad.Predicate)) subAnd.AddSubIterator(iterator.NewLinksTo(qs, all, quad.Subject)) hasa := iterator.NewHasA(qs, subAnd, quad.Object) and := iterator.NewAnd(qs) and.AddSubIterator(hasa) and.AddSubIterator(subIt) it = and case "has": fixed := qs.FixedIterator() if len(stringArgs) < 2 { return iterator.NewNull() } for _, name := range stringArgs[1:] { fixed.Add(qs.ValueOf(name)) } predFixed := qs.FixedIterator() predFixed.Add(qs.ValueOf(stringArgs[0])) subAnd := iterator.NewAnd(qs) subAnd.AddSubIterator(iterator.NewLinksTo(qs, predFixed, quad.Predicate)) subAnd.AddSubIterator(iterator.NewLinksTo(qs, fixed, quad.Object)) hasa := iterator.NewHasA(qs, subAnd, quad.Subject) and := iterator.NewAnd(qs) and.AddSubIterator(hasa) and.AddSubIterator(subIt) it = and case "morphism": it = base case "and": arg, _ := obj.Get("_gremlin_values") firstArg, _ := arg.Object().Get("0") if !isVertexChain(firstArg.Object()) { return iterator.NewNull() } argIt := buildIteratorTree(firstArg.Object(), qs) and := iterator.NewAnd(qs) and.AddSubIterator(subIt) and.AddSubIterator(argIt) it = and case "back": arg, _ := obj.Get("_gremlin_back_chain") argIt := buildIteratorTree(arg.Object(), qs) and := iterator.NewAnd(qs) and.AddSubIterator(subIt) and.AddSubIterator(argIt) it = and case "is": fixed := qs.FixedIterator() for _, name := range stringArgs { fixed.Add(qs.ValueOf(name)) } and := iterator.NewAnd(qs) and.AddSubIterator(fixed) and.AddSubIterator(subIt) it = and case "or": arg, _ := obj.Get("_gremlin_values") firstArg, _ := arg.Object().Get("0") if !isVertexChain(firstArg.Object()) { return iterator.NewNull() } argIt := buildIteratorTree(firstArg.Object(), qs) or := iterator.NewOr() or.AddSubIterator(subIt) or.AddSubIterator(argIt) it = or case "both": // Hardly the most efficient pattern, but the most general. // Worth looking into an Optimize() optimization here. clone := subIt.Clone() it1 := buildInOutIterator(obj, qs, subIt, false) it2 := buildInOutIterator(obj, qs, clone, true) or := iterator.NewOr() or.AddSubIterator(it1) or.AddSubIterator(it2) it = or case "out": it = buildInOutIterator(obj, qs, subIt, false) case "follow": // Follow a morphism arg, _ := obj.Get("_gremlin_values") firstArg, _ := arg.Object().Get("0") if isVertexChain(firstArg.Object()) { return iterator.NewNull() } it = buildIteratorTreeHelper(firstArg.Object(), qs, subIt) case "followr": // Follow a morphism arg, _ := obj.Get("_gremlin_followr") if isVertexChain(arg.Object()) { return iterator.NewNull() } it = buildIteratorTreeHelper(arg.Object(), qs, subIt) case "in": it = buildInOutIterator(obj, qs, subIt, true) case "except": arg, _ := obj.Get("_gremlin_values") firstArg, _ := arg.Object().Get("0") if !isVertexChain(firstArg.Object()) { return iterator.NewNull() } allIt := qs.NodesAllIterator() toComplementIt := buildIteratorTree(firstArg.Object(), qs) notIt := iterator.NewNot(toComplementIt, allIt) and := iterator.NewAnd(qs) and.AddSubIterator(subIt) and.AddSubIterator(notIt) it = and case "in_predicates": it = buildInOutPredicateIterator(obj, qs, subIt, true) case "out_predicates": it = buildInOutPredicateIterator(obj, qs, subIt, false) } if it == nil { panic("Iterator building does not catch the output iterator in some case.") } return it }
func (q *Query) buildIteratorTreeMapInternal(query map[string]interface{}, path Path) (graph.Iterator, error) { it := iterator.NewAnd(q.ses.qs) it.AddSubIterator(q.ses.qs.NodesAllIterator()) var err error err = nil outputStructure := make(map[string]interface{}) for key, subquery := range query { optional := false outputStructure[key] = nil reverse := false pred := key if strings.HasPrefix(pred, "@") { i := strings.Index(pred, ":") if i != -1 { pred = pred[(i + 1):] } } if strings.HasPrefix(pred, "!") { reverse = true pred = strings.TrimPrefix(pred, "!") } // Other special constructs here var subit graph.Iterator if key == "id" { subit, optional, err = q.buildIteratorTreeInternal(subquery, path.Follow(key)) if err != nil { return nil, err } } else { var builtIt graph.Iterator builtIt, optional, err = q.buildIteratorTreeInternal(subquery, path.Follow(key)) if err != nil { return nil, err } subAnd := iterator.NewAnd(q.ses.qs) predFixed := q.ses.qs.FixedIterator() predFixed.Add(q.ses.qs.ValueOf(pred)) subAnd.AddSubIterator(iterator.NewLinksTo(q.ses.qs, predFixed, quad.Predicate)) if reverse { lto := iterator.NewLinksTo(q.ses.qs, builtIt, quad.Subject) subAnd.AddSubIterator(lto) hasa := iterator.NewHasA(q.ses.qs, subAnd, quad.Object) subit = hasa } else { lto := iterator.NewLinksTo(q.ses.qs, builtIt, quad.Object) subAnd.AddSubIterator(lto) hasa := iterator.NewHasA(q.ses.qs, subAnd, quad.Subject) subit = hasa } } if optional { it.AddSubIterator(iterator.NewOptional(subit)) } else { it.AddSubIterator(subit) } } if err != nil { return nil, err } q.queryStructure[path] = outputStructure return it, nil }