func hasMorphism(via interface{}, nodes ...string) morphism { return morphism{ Name: "has", Reversal: func() morphism { return hasMorphism(via, nodes...) }, Apply: func(qs graph.QuadStore, it graph.Iterator) graph.Iterator { var sub graph.Iterator if len(nodes) == 0 { sub = qs.NodesAllIterator() } else { fixed := qs.FixedIterator() for _, n := range nodes { fixed.Add(qs.ValueOf(n)) } sub = fixed } var viaPath *Path if via != nil { viaPath = buildViaPath(qs, via) } else { viaPath = buildViaPath(qs) } subAnd := iterator.NewAnd(qs) subAnd.AddSubIterator(iterator.NewLinksTo(qs, sub, quad.Object)) subAnd.AddSubIterator(iterator.NewLinksTo(qs, viaPath.BuildIterator(), quad.Predicate)) hasa := iterator.NewHasA(qs, subAnd, quad.Subject) and := iterator.NewAnd(qs) and.AddSubIterator(it) and.AddSubIterator(hasa) return and }, } }
func TestIteratorsAndNextResultOrderA(t *testing.T) { glog.Info("\n-----------\n") inst, opts, err := createInstance() defer inst.Close() if err != nil { t.Fatalf("failed to create instance: %v", err) } qs, _, _ := makeTestStore(simpleGraph, opts) if qs.Size() != 11 { t.Fatal("Incorrect number of quads") } fixed := qs.FixedIterator() fixed.Add(qs.ValueOf("C")) fixed2 := qs.FixedIterator() fixed2.Add(qs.ValueOf("follows")) all := qs.NodesAllIterator() innerAnd := iterator.NewAnd(qs) innerAnd.AddSubIterator(iterator.NewLinksTo(qs, fixed2, quad.Predicate)) innerAnd.AddSubIterator(iterator.NewLinksTo(qs, all, quad.Object)) hasa := iterator.NewHasA(qs, innerAnd, quad.Subject) outerAnd := iterator.NewAnd(qs) outerAnd.AddSubIterator(fixed) outerAnd.AddSubIterator(hasa) if !outerAnd.Next() { t.Error("Expected one matching subtree") } val := outerAnd.Result() if qs.NameOf(val) != "C" { t.Errorf("Matching subtree should be %s, got %s", "barak", qs.NameOf(val)) } var ( got []string expect = []string{"B", "D"} ) for { got = append(got, qs.NameOf(all.Result())) if !outerAnd.NextPath() { break } } sort.Strings(got) if !reflect.DeepEqual(got, expect) { t.Errorf("Unexpected result, got:%q expect:%q", got, expect) } if outerAnd.Next() { t.Error("More than one possible top level output?") } }
func TestIteratorsAndNextResultOrderA(t *testing.T) { ts, _ := makeTestStore(simpleGraph) fixed := ts.FixedIterator() fixed.Add(ts.ValueOf("C")) fixed2 := ts.FixedIterator() fixed2.Add(ts.ValueOf("follows")) all := ts.NodesAllIterator() innerAnd := iterator.NewAnd() innerAnd.AddSubIterator(iterator.NewLinksTo(ts, fixed2, graph.Predicate)) innerAnd.AddSubIterator(iterator.NewLinksTo(ts, all, graph.Object)) hasa := iterator.NewHasA(ts, innerAnd, graph.Subject) outerAnd := iterator.NewAnd() outerAnd.AddSubIterator(fixed) outerAnd.AddSubIterator(hasa) val, ok := outerAnd.Next() if !ok { t.Error("Expected one matching subtree") } if ts.NameOf(val) != "C" { t.Errorf("Matching subtree should be %s, got %s", "barak", ts.NameOf(val)) } var ( got []string expect = []string{"B", "D"} ) for { got = append(got, ts.NameOf(all.Result())) if !outerAnd.NextResult() { break } } sort.Strings(got) if !reflect.DeepEqual(got, expect) { t.Errorf("Unexpected result, got:%q expect:%q", got, expect) } val, ok = outerAnd.Next() if ok { t.Error("More than one possible top level output?") } }
func (qs *QuadStore) optimizeAndIterator(it *iterator.And) (graph.Iterator, bool) { // Fail fast if nothing can happen glog.V(4).Infoln("Entering optimizeAndIterator", it.UID()) found := false for _, it := range it.SubIterators() { glog.V(4).Infoln(it.Type()) if it.Type() == mongoType { found = true } } if !found { glog.V(4).Infoln("Aborting optimizeAndIterator") return it, false } newAnd := iterator.NewAnd(qs) var mongoIt *Iterator for _, it := range it.SubIterators() { switch it.Type() { case mongoType: if mongoIt == nil { mongoIt = it.(*Iterator) } else { newAnd.AddSubIterator(it) } case graph.LinksTo: continue default: newAnd.AddSubIterator(it) } } stats := mongoIt.Stats() lset := []graph.Linkage{ { Dir: mongoIt.dir, Value: mongoIt.hash, }, } n := 0 for _, it := range it.SubIterators() { if it.Type() == graph.LinksTo { lto := it.(*iterator.LinksTo) // Is it more effective to do the replacement, or let the mongo check the linksto? ltostats := lto.Stats() if (ltostats.ContainsCost+stats.NextCost)*stats.Size > (ltostats.NextCost+stats.ContainsCost)*ltostats.Size { continue } newLto := NewLinksTo(qs, lto.SubIterators()[0], "quads", lto.Direction(), lset) newAnd.AddSubIterator(newLto) n++ } } if n == 0 { return it, false } return newAnd.Optimize() }
func TestRemoveQuad(t *testing.T) { qs, w, _ := makeTestStore(simpleGraph) w.RemoveQuad(quad.Quad{ Subject: "E", Predicate: "follows", Object: "F", Label: "", }) fixed := qs.FixedIterator() fixed.Add(qs.ValueOf("E")) fixed2 := qs.FixedIterator() fixed2.Add(qs.ValueOf("follows")) innerAnd := iterator.NewAnd() innerAnd.AddSubIterator(iterator.NewLinksTo(qs, fixed, quad.Subject)) innerAnd.AddSubIterator(iterator.NewLinksTo(qs, fixed2, quad.Predicate)) hasa := iterator.NewHasA(qs, innerAnd, quad.Object) newIt, _ := hasa.Optimize() if graph.Next(newIt) { t.Error("E should not have any followers.") } }
// join puts two iterators together by intersecting their result sets with an AND // Since we're using an and iterator, it's a good idea to put the smallest result // set first so that Next() produces fewer values to check Contains(). func join(qs graph.QuadStore, itL, itR graph.Iterator) graph.Iterator { and := iterator.NewAnd(qs) and.AddSubIterator(itL) and.AddSubIterator(itR) return and }
func TestRemoveQuad(t *testing.T) { qs, w, _ := makeTestStore(simpleGraph) err := w.RemoveQuad(quad.Make( "E", "follows", "F", "", )) if err != nil { t.Error("Couldn't remove quad", err) } fixed := qs.FixedIterator() fixed.Add(qs.ValueOf(quad.Raw("E"))) fixed2 := qs.FixedIterator() fixed2.Add(qs.ValueOf(quad.Raw("follows"))) innerAnd := iterator.NewAnd(qs) innerAnd.AddSubIterator(iterator.NewLinksTo(qs, fixed, quad.Subject)) innerAnd.AddSubIterator(iterator.NewLinksTo(qs, fixed2, quad.Predicate)) hasa := iterator.NewHasA(qs, innerAnd, quad.Object) newIt, _ := hasa.Optimize() if graph.Next(newIt) { t.Error("E should not have any followers.") } }
func TestIteratorsAndNextResultOrderA(t testing.TB, gen DatabaseFunc) { qs, opts, closer := gen(t) defer closer() MakeWriter(t, qs, opts, MakeQuadSet()...) require.Equal(t, int64(11), qs.Size(), "Incorrect number of quads") fixed := qs.FixedIterator() fixed.Add(qs.ValueOf(quad.Raw("C"))) fixed2 := qs.FixedIterator() fixed2.Add(qs.ValueOf(quad.Raw("follows"))) all := qs.NodesAllIterator() innerAnd := iterator.NewAnd(qs) innerAnd.AddSubIterator(iterator.NewLinksTo(qs, fixed2, quad.Predicate)) innerAnd.AddSubIterator(iterator.NewLinksTo(qs, all, quad.Object)) hasa := iterator.NewHasA(qs, innerAnd, quad.Subject) outerAnd := iterator.NewAnd(qs) outerAnd.AddSubIterator(fixed) outerAnd.AddSubIterator(hasa) require.True(t, outerAnd.Next(), "Expected one matching subtree") val := outerAnd.Result() require.Equal(t, quad.Raw("C"), qs.NameOf(val)) var ( got []string expect = []string{"B", "D"} ) for { got = append(got, qs.NameOf(all.Result()).String()) if !outerAnd.NextPath() { break } } sort.Strings(got) require.Equal(t, expect, got) require.True(t, !outerAnd.Next(), "More than one possible top level output?") }
func TestIteratorsAndNextResultOrderA(t *testing.T) { ts := MakeTestingMemstore() fixed := ts.FixedIterator() fixed.AddValue(ts.GetIdFor("C")) all := ts.GetNodesAllIterator() lto := iterator.NewLinksTo(ts, all, graph.Object) innerAnd := iterator.NewAnd() fixed2 := ts.FixedIterator() fixed2.AddValue(ts.GetIdFor("follows")) lto2 := iterator.NewLinksTo(ts, fixed2, graph.Predicate) innerAnd.AddSubIterator(lto2) innerAnd.AddSubIterator(lto) hasa := iterator.NewHasA(ts, innerAnd, graph.Subject) outerAnd := iterator.NewAnd() outerAnd.AddSubIterator(fixed) outerAnd.AddSubIterator(hasa) val, ok := outerAnd.Next() if !ok { t.Error("Expected one matching subtree") } if ts.GetNameFor(val) != "C" { t.Errorf("Matching subtree should be %s, got %s", "barak", ts.GetNameFor(val)) } expected := make([]string, 2) expected[0] = "B" expected[1] = "D" actualOut := make([]string, 2) actualOut[0] = ts.GetNameFor(all.LastResult()) nresultOk := outerAnd.NextResult() if !nresultOk { t.Error("Expected two results got one") } actualOut[1] = ts.GetNameFor(all.LastResult()) nresultOk = outerAnd.NextResult() if nresultOk { t.Error("Expected two results got three") } CompareStringSlices(t, expected, actualOut) val, ok = outerAnd.Next() if ok { t.Error("More than one possible top level output?") } }
// join puts two iterators together by intersecting their result sets with an AND // Since we're using an and iterator, it's a good idea to put the smallest result // set first so that Next() produces fewer values to check Contains(). func join(qs graph.QuadStore, its ...graph.Iterator) graph.Iterator { and := iterator.NewAnd(qs) for _, it := range its { if it == nil { continue } and.AddSubIterator(it) } return and }
func inOutIterator(viaPath *Path, it graph.Iterator, reverse bool) graph.Iterator { in, out := quad.Subject, quad.Object if reverse { in, out = out, in } lto := iterator.NewLinksTo(viaPath.qs, it, in) and := iterator.NewAnd(viaPath.qs) and.AddSubIterator(iterator.NewLinksTo(viaPath.qs, viaPath.BuildIterator(), quad.Predicate)) and.AddSubIterator(lto) return iterator.NewHasA(viaPath.qs, and, out) }
func iteratorMorphism(it graph.Iterator) morphism { return morphism{ Name: "iterator", Reversal: func() morphism { return iteratorMorphism(it) }, Apply: func(qs graph.QuadStore, subIt graph.Iterator) graph.Iterator { and := iterator.NewAnd(qs) and.AddSubIterator(it) and.AddSubIterator(subIt) return and }, } }
func andMorphism(p *Path) morphism { return morphism{ Name: "and", Reversal: func() morphism { return andMorphism(p) }, Apply: func(qs graph.QuadStore, it graph.Iterator) graph.Iterator { subIt := p.BuildIteratorOn(qs) and := iterator.NewAnd(qs) and.AddSubIterator(it) and.AddSubIterator(subIt) return and }, } }
func TestAndIterator(t *testing.T) { all1 := iterator.NewInt64(1, 3) all2 := iterator.NewInt64(3, 5) and := iterator.NewAnd() and.AddSubIterator(all1) and.AddSubIterator(all2) result := StringResultTreeEvaluator(and) expected := "(3 (3) (3))\n" if expected != result { t.Errorf("Expected %q got %q", expected, result) } }
func exceptMorphism(p *Path) morphism { return morphism{ Name: "except", Reversal: func() morphism { return exceptMorphism(p) }, Apply: func(qs graph.QuadStore, base graph.Iterator) graph.Iterator { subIt := p.BuildIteratorOn(qs) notIt := iterator.NewNot(subIt, qs.NodesAllIterator()) and := iterator.NewAnd(qs) and.AddSubIterator(base) and.AddSubIterator(notIt) return and }, } }
func buildSave(qs graph.QuadStore, via interface{}, tag string, it graph.Iterator, reverse bool) graph.Iterator { all := qs.NodesAllIterator() all.Tagger().Add(tag) node, allDir := quad.Subject, quad.Object var viaPath *Path if via != nil { viaPath = buildViaPath(qs, via) } else { viaPath = buildViaPath(qs) } if reverse { node, allDir = allDir, node } lto := iterator.NewLinksTo(qs, all, allDir) subAnd := iterator.NewAnd(qs) subAnd.AddSubIterator(iterator.NewLinksTo(qs, viaPath.BuildIterator(), quad.Predicate)) subAnd.AddSubIterator(lto) hasa := iterator.NewHasA(qs, subAnd, node) and := iterator.NewAnd(qs) and.AddSubIterator(hasa) and.AddSubIterator(it) return and }
func TestRemoveTriple(t *testing.T) { ts := MakeTestingMemstore() ts.RemoveTriple(&graph.Triple{"E", "follows", "F", ""}) fixed := ts.FixedIterator() fixed.AddValue(ts.GetIdFor("E")) lto := iterator.NewLinksTo(ts, fixed, graph.Subject) fixed2 := ts.FixedIterator() fixed2.AddValue(ts.GetIdFor("follows")) lto2 := iterator.NewLinksTo(ts, fixed2, graph.Predicate) innerAnd := iterator.NewAnd() innerAnd.AddSubIterator(lto2) innerAnd.AddSubIterator(lto) hasa := iterator.NewHasA(ts, innerAnd, graph.Object) newIt, _ := hasa.Optimize() _, ok := newIt.Next() if ok { t.Error("E should not have any followers.") } }
func buildInOutIterator(obj *otto.Object, ts graph.TripleStore, base graph.Iterator, isReverse bool) graph.Iterator { argList, _ := obj.Get("_gremlin_values") if argList.Class() != "GoArray" { glog.Errorln("How is arglist not an array? Return nothing.", argList.Class()) return iterator.NewNull() } argArray := argList.Object() lengthVal, _ := argArray.Get("length") length, _ := lengthVal.ToInteger() var predicateNodeIterator graph.Iterator if length == 0 { predicateNodeIterator = ts.NodesAllIterator() } else { zero, _ := argArray.Get("0") predicateNodeIterator = buildIteratorFromValue(zero, ts) } if length >= 2 { var tags []string one, _ := argArray.Get("1") if one.IsString() { s, _ := one.ToString() tags = append(tags, s) } else if one.Class() == "Array" { tags = makeListOfStringsFromArrayValue(one.Object()) } for _, tag := range tags { predicateNodeIterator.Tagger().Add(tag) } } in, out := quad.Subject, quad.Object if isReverse { in, out = out, in } lto := iterator.NewLinksTo(ts, base, in) and := iterator.NewAnd() and.AddSubIterator(iterator.NewLinksTo(ts, predicateNodeIterator, quad.Predicate)) and.AddSubIterator(lto) return iterator.NewHasA(ts, and, out) }
func (qs *QuadStore) optimizeAnd(it *iterator.And) (graph.Iterator, bool) { subs := it.SubIterators() var unusedIts []graph.Iterator var newit *SQLIterator newit = nil changed := false var err error for _, it := range subs { if it.Type() == sqlType { if newit == nil { newit = it.(*SQLIterator) } else { changed = true newit, err = intersect(newit.sql, it.(*SQLIterator).sql, qs) if err != nil { glog.Error(err) return it, false } } } else { unusedIts = append(unusedIts, it) } } if !changed { return it, false } if len(unusedIts) == 0 { newit.Tagger().CopyFrom(it) return newit, true } newAnd := iterator.NewAnd(qs) newAnd.Tagger().CopyFrom(it) newAnd.AddSubIterator(newit) for _, i := range unusedIts { newAnd.AddSubIterator(i) } return newAnd.Optimize() }
func isMorphism(nodes ...string) morphism { return morphism{ Name: "is", Reversal: func() morphism { return isMorphism(nodes...) }, Apply: func(qs graph.QuadStore, it graph.Iterator) graph.Iterator { var sub graph.Iterator if len(nodes) == 0 { sub = qs.NodesAllIterator() } else { fixed := qs.FixedIterator() for _, n := range nodes { fixed.Add(qs.ValueOf(n)) } sub = fixed } and := iterator.NewAnd(qs) and.AddSubIterator(sub) and.AddSubIterator(it) return and }, } }
func TestRemoveTriple(t *testing.T) { ts, _ := makeTestStore(simpleGraph) ts.RemoveTriple(&graph.Triple{"E", "follows", "F", ""}) fixed := ts.FixedIterator() fixed.Add(ts.ValueOf("E")) fixed2 := ts.FixedIterator() fixed2.Add(ts.ValueOf("follows")) innerAnd := iterator.NewAnd() innerAnd.AddSubIterator(iterator.NewLinksTo(ts, fixed, graph.Subject)) innerAnd.AddSubIterator(iterator.NewLinksTo(ts, fixed2, graph.Predicate)) hasa := iterator.NewHasA(ts, innerAnd, graph.Object) newIt, _ := hasa.Optimize() _, ok := newIt.Next() if ok { t.Error("E should not have any followers.") } }
func TestSetIterator(t *testing.T) { tmpDir, _ := ioutil.TempDir(os.TempDir(), "cayley_test") t.Log(tmpDir) defer os.RemoveAll(tmpDir) err := createNewLevelDB(tmpDir, nil) if err != nil { t.Fatalf("Failed to create working directory") } qs, err := newTripleStore(tmpDir, nil) if qs == nil || err != nil { t.Error("Failed to create leveldb TripleStore.") } defer qs.Close() qs.AddTripleSet(makeTripleSet()) expect := []*quad.Quad{ {"C", "follows", "B", ""}, {"C", "follows", "D", ""}, } sort.Sort(ordered(expect)) // Subject iterator. it := qs.TripleIterator(quad.Subject, qs.ValueOf("C")) if got := iteratedTriples(qs, it); !reflect.DeepEqual(got, expect) { t.Errorf("Failed to get expected results, got:%v expect:%v", got, expect) } it.Reset() and := iterator.NewAnd() and.AddSubIterator(qs.TriplesAllIterator()) and.AddSubIterator(it) if got := iteratedTriples(qs, and); !reflect.DeepEqual(got, expect) { t.Errorf("Failed to get confirm expected results, got:%v expect:%v", got, expect) } // Object iterator. it = qs.TripleIterator(quad.Object, qs.ValueOf("F")) expect = []*quad.Quad{ {"B", "follows", "F", ""}, {"E", "follows", "F", ""}, } sort.Sort(ordered(expect)) if got := iteratedTriples(qs, it); !reflect.DeepEqual(got, expect) { t.Errorf("Failed to get expected results, got:%v expect:%v", got, expect) } and = iterator.NewAnd() and.AddSubIterator(qs.TripleIterator(quad.Subject, qs.ValueOf("B"))) and.AddSubIterator(it) expect = []*quad.Quad{ {"B", "follows", "F", ""}, } if got := iteratedTriples(qs, and); !reflect.DeepEqual(got, expect) { t.Errorf("Failed to get confirm expected results, got:%v expect:%v", got, expect) } // Predicate iterator. it = qs.TripleIterator(quad.Predicate, qs.ValueOf("status")) expect = []*quad.Quad{ {"B", "status", "cool", "status_graph"}, {"D", "status", "cool", "status_graph"}, {"G", "status", "cool", "status_graph"}, } sort.Sort(ordered(expect)) if got := iteratedTriples(qs, it); !reflect.DeepEqual(got, expect) { t.Errorf("Failed to get expected results from predicate iterator, got:%v expect:%v", got, expect) } // Label iterator. it = qs.TripleIterator(quad.Label, qs.ValueOf("status_graph")) expect = []*quad.Quad{ {"B", "status", "cool", "status_graph"}, {"D", "status", "cool", "status_graph"}, {"G", "status", "cool", "status_graph"}, } sort.Sort(ordered(expect)) if got := iteratedTriples(qs, it); !reflect.DeepEqual(got, expect) { t.Errorf("Failed to get expected results from predicate iterator, got:%v expect:%v", got, expect) } it.Reset() // Order is important and = iterator.NewAnd() and.AddSubIterator(qs.TripleIterator(quad.Subject, qs.ValueOf("B"))) and.AddSubIterator(it) expect = []*quad.Quad{ {"B", "status", "cool", "status_graph"}, } if got := iteratedTriples(qs, and); !reflect.DeepEqual(got, expect) { t.Errorf("Failed to get confirm expected results, got:%v expect:%v", got, expect) } it.Reset() // Order is important and = iterator.NewAnd() and.AddSubIterator(it) and.AddSubIterator(qs.TripleIterator(quad.Subject, qs.ValueOf("B"))) expect = []*quad.Quad{ {"B", "status", "cool", "status_graph"}, } if got := iteratedTriples(qs, and); !reflect.DeepEqual(got, expect) { t.Errorf("Failed to get confirm expected results, got:%v expect:%v", got, expect) } }
// TestQuadStoreQuadIteratorAnd tests the QuadIterator method of a // graph.QuadStore by issuing several queries against a fixture. func TestQuadStoreQuadIteratorAnd(t *testing.T, ctx context.Context) { qs := ContextQuadStore(ctx) _, err := WriteFixtureQuadStore(qs, "simple") if err != nil { t.Errorf("Unexpected error writing fixures: %v", err) } var tests = []struct { dir quad.Direction name string anddir quad.Direction andname string expect []quad.Quad }{ { quad.Subject, "C", quad.Any, "", []quad.Quad{ {"C", "follows", "B", ""}, {"C", "follows", "D", ""}, }, }, { quad.Object, "F", quad.Subject, "B", []quad.Quad{ {"B", "follows", "F", ""}, }, }, { quad.Predicate, "status", quad.Subject, "G", []quad.Quad{ {"G", "status", "cool", "status_graph"}, }, }, { quad.Label, "status_graph", quad.Subject, "B", []quad.Quad{ {"B", "status", "cool", "status_graph"}, }, }, } for i, test := range tests { func() { it := qs.QuadIterator(test.dir, qs.ValueOf(test.name)) defer it.Reset() and := iterator.NewAnd(qs) var other graph.Iterator if test.anddir == quad.Any { other = qs.QuadsAllIterator() } else { other = qs.QuadIterator(test.anddir, qs.ValueOf(test.andname)) } defer other.Reset() and.AddSubIterator(other) and.AddSubIterator(it) defer and.Reset() quads := IterateQuads(qs, and) if !reflect.DeepEqual(quads, test.expect) { t.Errorf("Test %d: Failed to get expected results, got:%q expect:%q", i, quads, test.expect) } }() } }
func buildIteratorTreeHelper(obj *otto.Object, ts graph.TripleStore, base graph.Iterator) graph.Iterator { var it graph.Iterator it = base // TODO: Better error handling kindVal, _ := obj.Get("_gremlin_type") stringArgs := getStringArgs(obj) var subIt graph.Iterator prevVal, _ := obj.Get("_gremlin_prev") if !prevVal.IsObject() { subIt = base } else { subIt = buildIteratorTreeHelper(prevVal.Object(), ts, base) } kind, _ := kindVal.ToString() switch kind { case "vertex": if len(stringArgs) == 0 { it = ts.NodesAllIterator() } else { fixed := ts.FixedIterator() for _, name := range stringArgs { fixed.Add(ts.ValueOf(name)) } it = fixed } case "tag": it = subIt for _, tag := range stringArgs { it.Tagger().Add(tag) } case "save": all := ts.NodesAllIterator() if len(stringArgs) > 2 || len(stringArgs) == 0 { return iterator.NewNull() } if len(stringArgs) == 2 { all.Tagger().Add(stringArgs[1]) } else { all.Tagger().Add(stringArgs[0]) } predFixed := ts.FixedIterator() predFixed.Add(ts.ValueOf(stringArgs[0])) subAnd := iterator.NewAnd() subAnd.AddSubIterator(iterator.NewLinksTo(ts, predFixed, quad.Predicate)) subAnd.AddSubIterator(iterator.NewLinksTo(ts, all, quad.Object)) hasa := iterator.NewHasA(ts, subAnd, quad.Subject) and := iterator.NewAnd() and.AddSubIterator(hasa) and.AddSubIterator(subIt) it = and case "saver": all := ts.NodesAllIterator() if len(stringArgs) > 2 || len(stringArgs) == 0 { return iterator.NewNull() } if len(stringArgs) == 2 { all.Tagger().Add(stringArgs[1]) } else { all.Tagger().Add(stringArgs[0]) } predFixed := ts.FixedIterator() predFixed.Add(ts.ValueOf(stringArgs[0])) subAnd := iterator.NewAnd() subAnd.AddSubIterator(iterator.NewLinksTo(ts, predFixed, quad.Predicate)) subAnd.AddSubIterator(iterator.NewLinksTo(ts, all, quad.Subject)) hasa := iterator.NewHasA(ts, subAnd, quad.Object) and := iterator.NewAnd() and.AddSubIterator(hasa) and.AddSubIterator(subIt) it = and case "has": fixed := ts.FixedIterator() if len(stringArgs) < 2 { return iterator.NewNull() } for _, name := range stringArgs[1:] { fixed.Add(ts.ValueOf(name)) } predFixed := ts.FixedIterator() predFixed.Add(ts.ValueOf(stringArgs[0])) subAnd := iterator.NewAnd() subAnd.AddSubIterator(iterator.NewLinksTo(ts, predFixed, quad.Predicate)) subAnd.AddSubIterator(iterator.NewLinksTo(ts, fixed, quad.Object)) hasa := iterator.NewHasA(ts, subAnd, quad.Subject) and := iterator.NewAnd() and.AddSubIterator(hasa) and.AddSubIterator(subIt) it = and case "morphism": it = base case "and": arg, _ := obj.Get("_gremlin_values") firstArg, _ := arg.Object().Get("0") if !isVertexChain(firstArg.Object()) { return iterator.NewNull() } argIt := buildIteratorTree(firstArg.Object(), ts) and := iterator.NewAnd() and.AddSubIterator(subIt) and.AddSubIterator(argIt) it = and case "back": arg, _ := obj.Get("_gremlin_back_chain") argIt := buildIteratorTree(arg.Object(), ts) and := iterator.NewAnd() and.AddSubIterator(subIt) and.AddSubIterator(argIt) it = and case "is": fixed := ts.FixedIterator() for _, name := range stringArgs { fixed.Add(ts.ValueOf(name)) } and := iterator.NewAnd() and.AddSubIterator(fixed) and.AddSubIterator(subIt) it = and case "or": arg, _ := obj.Get("_gremlin_values") firstArg, _ := arg.Object().Get("0") if !isVertexChain(firstArg.Object()) { return iterator.NewNull() } argIt := buildIteratorTree(firstArg.Object(), ts) or := iterator.NewOr() or.AddSubIterator(subIt) or.AddSubIterator(argIt) it = or case "both": // Hardly the most efficient pattern, but the most general. // Worth looking into an Optimize() optimization here. clone := subIt.Clone() it1 := buildInOutIterator(obj, ts, subIt, false) it2 := buildInOutIterator(obj, ts, clone, true) or := iterator.NewOr() or.AddSubIterator(it1) or.AddSubIterator(it2) it = or case "out": it = buildInOutIterator(obj, ts, subIt, false) case "follow": // Follow a morphism arg, _ := obj.Get("_gremlin_values") firstArg, _ := arg.Object().Get("0") if isVertexChain(firstArg.Object()) { return iterator.NewNull() } it = buildIteratorTreeHelper(firstArg.Object(), ts, subIt) case "followr": // Follow a morphism arg, _ := obj.Get("_gremlin_followr") if isVertexChain(arg.Object()) { return iterator.NewNull() } it = buildIteratorTreeHelper(arg.Object(), ts, subIt) case "in": it = buildInOutIterator(obj, ts, subIt, true) } return it }
func TestSetIterator(t *testing.T) { var ts *TripleStore var tmpDir string Convey("Given a prepared database", t, func() { tmpDir, _ = ioutil.TempDir(os.TempDir(), "cayley_test") t.Log(tmpDir) defer os.RemoveAll(tmpDir) ok := CreateNewLevelDB(tmpDir) So(ok, ShouldBeTrue) ts = NewTripleStore(tmpDir, nil) ts.AddTripleSet(makeTripleSet()) var it graph.Iterator Convey("Can create a subject iterator", func() { it = ts.GetTripleIterator(graph.Subject, ts.GetIdFor("C")) Convey("Containing the right things", func() { expected := []string{ (&graph.Triple{"C", "follows", "B", ""}).String(), (&graph.Triple{"C", "follows", "D", ""}).String(), } actual := extractTripleFromIterator(ts, it) sort.Strings(actual) sort.Strings(expected) So(actual, ShouldResemble, expected) }) Convey("And checkable", func() { and := iterator.NewAnd() and.AddSubIterator(ts.GetTriplesAllIterator()) and.AddSubIterator(it) expected := []string{ (&graph.Triple{"C", "follows", "B", ""}).String(), (&graph.Triple{"C", "follows", "D", ""}).String(), } actual := extractTripleFromIterator(ts, and) sort.Strings(actual) sort.Strings(expected) So(actual, ShouldResemble, expected) }) Reset(func() { it.Reset() }) }) Convey("Can create an object iterator", func() { it = ts.GetTripleIterator(graph.Object, ts.GetIdFor("F")) Convey("Containing the right things", func() { expected := []string{ (&graph.Triple{"B", "follows", "F", ""}).String(), (&graph.Triple{"E", "follows", "F", ""}).String(), } actual := extractTripleFromIterator(ts, it) sort.Strings(actual) sort.Strings(expected) So(actual, ShouldResemble, expected) }) Convey("Mutually and-checkable", func() { and := iterator.NewAnd() and.AddSubIterator(ts.GetTripleIterator(graph.Subject, ts.GetIdFor("B"))) and.AddSubIterator(it) expected := []string{ (&graph.Triple{"B", "follows", "F", ""}).String(), } actual := extractTripleFromIterator(ts, and) sort.Strings(actual) sort.Strings(expected) So(actual, ShouldResemble, expected) }) }) Convey("Can create a predicate iterator", func() { it = ts.GetTripleIterator(graph.Predicate, ts.GetIdFor("status")) Convey("Containing the right things", func() { expected := []string{ (&graph.Triple{"B", "status", "cool", "status_graph"}).String(), (&graph.Triple{"D", "status", "cool", "status_graph"}).String(), (&graph.Triple{"G", "status", "cool", "status_graph"}).String(), } actual := extractTripleFromIterator(ts, it) sort.Strings(actual) sort.Strings(expected) So(actual, ShouldResemble, expected) }) }) Convey("Can create a provenance iterator", func() { it = ts.GetTripleIterator(graph.Provenance, ts.GetIdFor("status_graph")) Convey("Containing the right things", func() { expected := []string{ (&graph.Triple{"B", "status", "cool", "status_graph"}).String(), (&graph.Triple{"D", "status", "cool", "status_graph"}).String(), (&graph.Triple{"G", "status", "cool", "status_graph"}).String(), } actual := extractTripleFromIterator(ts, it) sort.Strings(actual) sort.Strings(expected) So(actual, ShouldResemble, expected) }) Convey("Can be cross-checked", func() { and := iterator.NewAnd() // Order is important and.AddSubIterator(ts.GetTripleIterator(graph.Subject, ts.GetIdFor("B"))) and.AddSubIterator(it) expected := []string{ (&graph.Triple{"B", "status", "cool", "status_graph"}).String(), } actual := extractTripleFromIterator(ts, and) So(actual, ShouldResemble, expected) }) Convey("Can check against other iterators", func() { and := iterator.NewAnd() // Order is important and.AddSubIterator(it) and.AddSubIterator(ts.GetTripleIterator(graph.Subject, ts.GetIdFor("B"))) expected := []string{ (&graph.Triple{"B", "status", "cool", "status_graph"}).String(), } actual := extractTripleFromIterator(ts, and) So(actual, ShouldResemble, expected) }) Reset(func() { it.Reset() }) }) Reset(func() { ts.Close() }) }) }
func (q *Query) buildIteratorTreeMapInternal(query map[string]interface{}, path Path) (graph.Iterator, error) { it := iterator.NewAnd(q.ses.qs) it.AddSubIterator(q.ses.qs.NodesAllIterator()) var err error err = nil outputStructure := make(map[string]interface{}) for key, subquery := range query { optional := false outputStructure[key] = nil reverse := false pred := key if strings.HasPrefix(pred, "@") { i := strings.Index(pred, ":") if i != -1 { pred = pred[(i + 1):] } } if strings.HasPrefix(pred, "!") { reverse = true pred = strings.TrimPrefix(pred, "!") } // Other special constructs here var subit graph.Iterator if key == "id" { subit, optional, err = q.buildIteratorTreeInternal(subquery, path.Follow(key)) if err != nil { return nil, err } } else { var builtIt graph.Iterator builtIt, optional, err = q.buildIteratorTreeInternal(subquery, path.Follow(key)) if err != nil { return nil, err } subAnd := iterator.NewAnd(q.ses.qs) predFixed := q.ses.qs.FixedIterator() predFixed.Add(q.ses.qs.ValueOf(quad.Raw(pred))) subAnd.AddSubIterator(iterator.NewLinksTo(q.ses.qs, predFixed, quad.Predicate)) if reverse { lto := iterator.NewLinksTo(q.ses.qs, builtIt, quad.Subject) subAnd.AddSubIterator(lto) hasa := iterator.NewHasA(q.ses.qs, subAnd, quad.Object) subit = hasa } else { lto := iterator.NewLinksTo(q.ses.qs, builtIt, quad.Object) subAnd.AddSubIterator(lto) hasa := iterator.NewHasA(q.ses.qs, subAnd, quad.Subject) subit = hasa } } if optional { it.AddSubIterator(iterator.NewOptional(subit)) } else { it.AddSubIterator(subit) } } if err != nil { return nil, err } q.queryStructure[path] = outputStructure return it, nil }
func TestSetIterator(t *testing.T) { tmpDir, _ := ioutil.TempDir(os.TempDir(), "cayley_test") t.Log(tmpDir) defer os.RemoveAll(tmpDir) ok := CreateNewLevelDB(tmpDir) if !ok { t.Fatalf("Failed to create working directory") } ts := NewTripleStore(tmpDir, nil) defer ts.Close() ts.AddTripleSet(makeTripleSet()) expect := []*graph.Triple{ {"C", "follows", "B", ""}, {"C", "follows", "D", ""}, } sort.Sort(ordered(expect)) // Subject iterator. it := ts.TripleIterator(graph.Subject, ts.ValueOf("C")) if got := iteratedTriples(ts, it); !reflect.DeepEqual(got, expect) { t.Errorf("Failed to get expected results, got:%v expect:%v", got, expect) } it.Reset() and := iterator.NewAnd() and.AddSubIterator(ts.TriplesAllIterator()) and.AddSubIterator(it) if got := iteratedTriples(ts, and); !reflect.DeepEqual(got, expect) { t.Errorf("Failed to get confirm expected results, got:%v expect:%v", got, expect) } // Object iterator. it = ts.TripleIterator(graph.Object, ts.ValueOf("F")) expect = []*graph.Triple{ {"B", "follows", "F", ""}, {"E", "follows", "F", ""}, } sort.Sort(ordered(expect)) if got := iteratedTriples(ts, it); !reflect.DeepEqual(got, expect) { t.Errorf("Failed to get expected results, got:%v expect:%v", got, expect) } and = iterator.NewAnd() and.AddSubIterator(ts.TripleIterator(graph.Subject, ts.ValueOf("B"))) and.AddSubIterator(it) expect = []*graph.Triple{ {"B", "follows", "F", ""}, } if got := iteratedTriples(ts, and); !reflect.DeepEqual(got, expect) { t.Errorf("Failed to get confirm expected results, got:%v expect:%v", got, expect) } // Predicate iterator. it = ts.TripleIterator(graph.Predicate, ts.ValueOf("status")) expect = []*graph.Triple{ {"B", "status", "cool", "status_graph"}, {"D", "status", "cool", "status_graph"}, {"G", "status", "cool", "status_graph"}, } sort.Sort(ordered(expect)) if got := iteratedTriples(ts, it); !reflect.DeepEqual(got, expect) { t.Errorf("Failed to get expected results from predicate iterator, got:%v expect:%v", got, expect) } // Provenance iterator. it = ts.TripleIterator(graph.Provenance, ts.ValueOf("status_graph")) expect = []*graph.Triple{ {"B", "status", "cool", "status_graph"}, {"D", "status", "cool", "status_graph"}, {"G", "status", "cool", "status_graph"}, } sort.Sort(ordered(expect)) if got := iteratedTriples(ts, it); !reflect.DeepEqual(got, expect) { t.Errorf("Failed to get expected results from predicate iterator, got:%v expect:%v", got, expect) } it.Reset() // Order is important and = iterator.NewAnd() and.AddSubIterator(ts.TripleIterator(graph.Subject, ts.ValueOf("B"))) and.AddSubIterator(it) expect = []*graph.Triple{ {"B", "status", "cool", "status_graph"}, } if got := iteratedTriples(ts, and); !reflect.DeepEqual(got, expect) { t.Errorf("Failed to get confirm expected results, got:%v expect:%v", got, expect) } it.Reset() // Order is important and = iterator.NewAnd() and.AddSubIterator(it) and.AddSubIterator(ts.TripleIterator(graph.Subject, ts.ValueOf("B"))) expect = []*graph.Triple{ {"B", "status", "cool", "status_graph"}, } if got := iteratedTriples(ts, and); !reflect.DeepEqual(got, expect) { t.Errorf("Failed to get confirm expected results, got:%v expect:%v", got, expect) } }
func (qs *QuadStore) optimizeAnd(it *iterator.And) (graph.Iterator, bool) { subs := it.SubIterators() var unusedIts []graph.Iterator var newit *SQLIterator newit = nil changed := false var err error // Combine SQL iterators glog.V(4).Infof("Combining SQL %#v", subs) for _, subit := range subs { if subit.Type() == sqlType { if newit == nil { newit = subit.(*SQLIterator) } else { changed = true newit, err = intersect(newit.sql, subit.(*SQLIterator).sql, qs) if err != nil { glog.Error(err) return it, false } } } else { unusedIts = append(unusedIts, subit) } } if newit == nil { return it, false } // Combine fixed iterators into the SQL iterators. glog.V(4).Infof("Combining fixed %#v", unusedIts) var nodeit *SQLNodeIterator if n, ok := newit.sql.(*SQLNodeIterator); ok { nodeit = n } else if n, ok := newit.sql.(*SQLNodeIntersection); ok { nodeit = n.nodeIts[0].(*SQLNodeIterator) } if nodeit != nil { passOneIts := unusedIts unusedIts = nil for _, subit := range passOneIts { if subit.Type() != graph.Fixed { unusedIts = append(unusedIts, subit) continue } changed = true for graph.Next(subit) { nodeit.fixedSet = append(nodeit.fixedSet, qs.NameOf(subit.Result())) } } } if !changed { return it, false } // Clean up if we're done. if len(unusedIts) == 0 { newit.Tagger().CopyFrom(it) return newit, true } newAnd := iterator.NewAnd(qs) newAnd.Tagger().CopyFrom(it) newAnd.AddSubIterator(newit) for _, i := range unusedIts { newAnd.AddSubIterator(i) } return newAnd.Optimize() }
func buildIteratorTree(tree *peg.ExpressionTree, qs graph.QuadStore) graph.Iterator { switch tree.Name { case "Start": return buildIteratorTree(tree.Children[0], qs) case "NodeIdentifier": var out graph.Iterator nodeID := getIdentString(tree) if tree.Children[0].Name == "Variable" { allIt := qs.NodesAllIterator() allIt.Tagger().Add(nodeID) out = allIt } else { n := nodeID if tree.Children[0].Children[0].Name == "ColonIdentifier" { n = nodeID[1:] } fixed := qs.FixedIterator() fixed.Add(qs.ValueOf(n)) out = fixed } return out case "PredIdentifier": i := 0 if tree.Children[0].Name == "Reverse" { //Taken care of below i++ } it := buildIteratorTree(tree.Children[i], qs) lto := iterator.NewLinksTo(qs, it, quad.Predicate) return lto case "RootConstraint": constraintCount := 0 and := iterator.NewAnd() for _, c := range tree.Children { switch c.Name { case "NodeIdentifier": fallthrough case "Constraint": it := buildIteratorTree(c, qs) and.AddSubIterator(it) constraintCount++ continue default: continue } } return and case "Constraint": var hasa *iterator.HasA topLevelDir := quad.Subject subItDir := quad.Object subAnd := iterator.NewAnd() isOptional := false for _, c := range tree.Children { switch c.Name { case "PredIdentifier": if c.Children[0].Name == "Reverse" { topLevelDir = quad.Object subItDir = quad.Subject } it := buildIteratorTree(c, qs) subAnd.AddSubIterator(it) continue case "PredicateKeyword": switch c.Children[0].Name { case "OptionalKeyword": isOptional = true } case "NodeIdentifier": fallthrough case "RootConstraint": it := buildIteratorTree(c, qs) l := iterator.NewLinksTo(qs, it, subItDir) subAnd.AddSubIterator(l) continue default: continue } } hasa = iterator.NewHasA(qs, subAnd, topLevelDir) if isOptional { optional := iterator.NewOptional(hasa) return optional } return hasa default: return &iterator.Null{} } panic("Not reached") }
func TestSetIterator(t testing.TB, gen DatabaseFunc) { qs, opts, closer := gen(t) defer closer() MakeWriter(t, qs, opts, MakeQuadSet()...) expectIteratedQuads := func(it graph.Iterator, exp []quad.Quad) { ExpectIteratedQuads(t, qs, it, exp) } // Subject iterator. it := qs.QuadIterator(quad.Subject, qs.ValueOf(quad.Raw("C"))) expectIteratedQuads(it, []quad.Quad{ quad.Make("C", "follows", "B", ""), quad.Make("C", "follows", "D", ""), }) it.Reset() and := iterator.NewAnd(qs) and.AddSubIterator(qs.QuadsAllIterator()) and.AddSubIterator(it) expectIteratedQuads(and, []quad.Quad{ quad.Make("C", "follows", "B", ""), quad.Make("C", "follows", "D", ""), }) // Object iterator. it = qs.QuadIterator(quad.Object, qs.ValueOf(quad.Raw("F"))) expectIteratedQuads(it, []quad.Quad{ quad.Make("B", "follows", "F", ""), quad.Make("E", "follows", "F", ""), }) and = iterator.NewAnd(qs) and.AddSubIterator(qs.QuadIterator(quad.Subject, qs.ValueOf(quad.Raw("B")))) and.AddSubIterator(it) expectIteratedQuads(and, []quad.Quad{ quad.Make("B", "follows", "F", ""), }) // Predicate iterator. it = qs.QuadIterator(quad.Predicate, qs.ValueOf(quad.Raw("status"))) expectIteratedQuads(it, []quad.Quad{ quad.Make("B", "status", "cool", "status_graph"), quad.Make("D", "status", "cool", "status_graph"), quad.Make("G", "status", "cool", "status_graph"), }) // Label iterator. it = qs.QuadIterator(quad.Label, qs.ValueOf(quad.Raw("status_graph"))) expectIteratedQuads(it, []quad.Quad{ quad.Make("B", "status", "cool", "status_graph"), quad.Make("D", "status", "cool", "status_graph"), quad.Make("G", "status", "cool", "status_graph"), }) it.Reset() // Order is important and = iterator.NewAnd(qs) and.AddSubIterator(qs.QuadIterator(quad.Subject, qs.ValueOf(quad.Raw("B")))) and.AddSubIterator(it) expectIteratedQuads(and, []quad.Quad{ quad.Make("B", "status", "cool", "status_graph"), }) it.Reset() // Order is important and = iterator.NewAnd(qs) and.AddSubIterator(it) and.AddSubIterator(qs.QuadIterator(quad.Subject, qs.ValueOf(quad.Raw("B")))) expectIteratedQuads(and, []quad.Quad{ quad.Make("B", "status", "cool", "status_graph"), }) }